OP YANBU.
Your XP cannot dictate where you live nor stop you moving and no judge would intervene in your situation. It's a no brainer for you to live where there is support for you and DBaby, as you will be doing majority of care for your baby.
Also you don't have to do the travel to facilitate contact. Not being able to afford it, is a sound reason in this case. As is how tiring it is when you are doing all the waking nights.
He doesn't sound a hands on dad. He WILL learn, as you won't be there to do it for him when DBaby is old enough (and not ebf) to go for longer periods. So that will be lovely to have a break which you haven't had so far.
Your XP doesn't get to dictate to you where you live and expect you to drive baby there and back for 2 hours several times a week (especially not at a time after work that suits him through rush hour, & after babies bedtime!)
It's nice of you offer to do pick up (don't offer drop off, take it from me). Only you know how reliable he will be and whether you want to share travel or meet halfway etc. It's always good to work together if you can but it won't be court ordered if you have good reason not to. If you'd moved 4 hours away then maybe it'd be fairer to share.
Given his long working hours, he'll get EOW when proper overnight contact starts as I doubt he could do midweek as well given his late working hours and well... young baby. Right now he might want to travel down to you & do short daytime contact.
My XH moved 5 mins away, then moved twice more, 1 hour away (2 hours in rush hour on motorway), then again so he was 3 hours away. He tried to get judge to force me to drive DCs for travel (I can't drive that far, spinal condition... and even without that...) Judge told him he was being unreasonable- he said "MrBedhampron you have 12 days off a fortnight so to expect an exhausted RP mum to drive 6 hours there and back all day Sunday into the evening when she gets he.to then have dinner to start to cook for.3 DC, homework and preparations for school & nursery to make for Monday morning as well as her work and children's routines & baths to settle into bed, is not in my opinion a reasonable demand and I will not order such".
Judge told my XH if he wanted contact he would make suitable travel arrangements himself. Judges differ but my XH got told off repeatedly by different judges as he was litigious and awkward. I facilitate contact but I am not anyone's dogsbody (except my DCs!)
Really tho' an hour isn't much and if you are kind enough to share pick ups, that would be a good thing to do. But right now your XP needs to do driving to you whilst you are ebf -ing and whilst it is only a short contact.