I feel you OP, me and my partner live a 45 minute car journey apart (not sure about the travel time on public transport as due to health issues i can't use it) and because of MH issues on his end, he can't travel at all if he's not walking there (panic attacks in any kind of vehicle).
So, once a week, my dad drives me to his (i can't drive myself due to health issues) and picks me up the next day. This has been the set up for almost 3 years, and while i completely understand he's physically incapable of coming to me, or even meeting in the middle as he literally cannot leave the little town he lives in (which is truly awful for him) it gets exhausting always being the one doing the travelling.
I am completely in love with him and want to spend the rest of my life with him, and it's just taking time to work things out to live together, and i miss him like crazy when we're apart. But, some weeks i get a feeling of dread about having to do the travelling, and put off going until a later day in the week, and have even felt relief occasionally if my dad has plans and can't take me.
My partner has never been anything but grateful to me for doing the travel to allow us to be together, and to my dad for being willing to facilitate it. If he made a comment like your partner did, it would hurt a lot, and leave a nasty taste.
In your shoes where your partner is able to travel, i'd get this planned visit over with, then when planning the next get together, i wouldn't agree to any plan that isn't him coming at LEAST half way, if not fully, to you. And from there, alternate who is the one travelling. That is of course if this hasn't already put you off him, as relationships with a distance are more difficult to maintain and grow. Ultimately it may be better to call time now a couple of months in before it gets more serious with stronger feelings.