I'd say no, don't do it.
THere is literally no benefit to you to move - all the benefit is to him.
And if he's going to leave you on your own more to pursue his own hobbies and friendships, then what the hell are you going to be doing - well, looking after baby on your own, obviously! With no support.
NO benefit to you at all.
If you think your marriage can take it, I would actually suggest he spends every other weekend in London, doing his thing, with his mates. That way he gets his wants met, but you don't have to lose all your benefits of living where you love.
Since he actively wants to spend less time with you, this will probably appeal to him - might not to you, but at least you get to stay with friends and family!
And in all honesty, he doesn't sound very committed to your marriage - who on earth admits to their wife and the mother of their child that they want to spend less time with them?! That's so wrong. So I think that even if you do move "for him" he's still going to end up leaving you in the lurch, in a place you don't know, with no support.
Stay where you are. Offer him more time in London to pursue his interests, but don't move.