How long have you been giving him a ride to work?
Is it impossible for him to get there w/out public transportation?

It just seems like he would eventually.. feel bad about putting you out (especially since he lives rent free). Why doesnt he pay rent? What would he be doing if you werent there to support him i guess is what im asking.

(How old are you guys?)

Regardless of all you do for him i dont think you can hold that over him when it comes to coming home early to spend time with you. You yourself said you guys spend most of your free time together.
I think its separate issues.
You dont do all these things for him because thats how you make him spend time with you right?
So i think that should be a different argument. Because your other points are really valid and you dont want the whole "you owe it to me to spend time with me" to undermine the validity of everything else.