It's not about how your XH behaved when he left you, though - it's the here and now, the relationship you want your child to have with his father. And he may say he would move for work - but he hasn't.
It's all very well for other posters to bring up military fathers... but that's not ideal. Plenty of kids do just fine without a dad at all but let's not have a race to the bottom.
Like a PP, yes I'm bringing my own history to this. Don't we all?
My XH is an arsehole, turned out he fucked prostitutes our entire relationship. Possibly in the fancy hotel the night before our wedding, which we then shared on our wedding night
He was utterly shit with our small child - really lazy, did nothing with her if I was around to do it.
I'd moved 3 hours away to his town, have a killer commute and - if I don't leave my company - am forced by location to tread water career wise. That said, I'm treading water in warm shallows - I like my job and am paid well. But the mammoth commute fucks my personal life about big style.
With one phone call to my boss, I could have relocated with my daughter, doubled my salary with a relocation abroad and had a hefty chunk of school fees paid for her. And got a nice chunk of maintenance from a largely absent father. And had more time in my personal life and more career opportunities.
I chose to stay.
My daughter now has a great relationship with her dad - partly because he had to step up with having her, mostly because he got a GF with a daughter and turned Disney Dad (like, actually taking her to Disney for a fortnight when previous summer he's managed 2 night holiday
when he wasn't trying to impress the GF) I digress...
All of this - I'm not a martyr, I'm happy here too. I haven't trashed my life for her relationship with him. But I definitely prioritised her long term relationship with her father over me, and my chances of meeting someone with my resulting lack of free time. I'm glad I did. (and I have met someone who loves me enough to work with my time constraints)
I just think you have to think REALLY carefully before taking away regular and frequent contact between a parent and child. And as I said before - I'd go to court to block you.