runny - Was this a first date?
It's a bit poor form to expect you to travel to him. I can't remember if you've met him before? Presumably not if you're unsure whether it'll be a bad date or not.
If there's no realistic reason to think that you'd be able to fit a relationship with him into your life, then there's not much point meeting him, is there? It's one thing if you just fancy a one-off trip into town, but..
If it's a first date he should really make some effort to meet you halfway, or come to your neck of the woods. In my opinion anyway.
I've got the same dilemma with my date for this weekend. She's over an hours drive away from home and work, public transport is a sod to get there, and I'd like to have a drink on a first date as coffee dates are so sterile - especially with three hours of driving involved too.
So I'm getting an Airbnb in her town. We know we get on well by phone, so it's unlikely to be tumbleweed on the night, and in future I can drive there and back sometimes. But not all the time.