My nieces live in a town where neither of their divorced parents particularly want to live. The mum wants to move back to her very small rural town, the dad wants to move to Seattle. He could not get work in the rural town and has no links to it, she could not get a visa to go to Seattle although he would have hugely enhanced job opportunities there, so they stay where they are for the children's sakes - it is the best compromise location.
They both recognise and agree this is a sacrifice they will make till the youngest is 16, and then able to travel freely. It is tough, and unfair, of course it is. But it is also the right thing to do.
I had a good high school friend whose father moved far away when he was 10. He still gets teary-eyed thinking about it. He feels his relationship never really got to grow past that point. Even now, when we are an age when our parents are elderly and some have passed on, he mused how we all knew each other's parents and attended their funerals, but no-one knew his dad at all and so when the time comes he will have less support around.
My own father left my mum and moved 10,000 miles away. My ILs barely know him, whereas my mother is completely integrated into their family unit. This makes me sad in many ways, as I feel that they only half know me. And I was in my twenties when he moved!
The other thing to think about is what choices your children will make later. My friend's mum moved from India to UK, taking the girls with her. One later decided to move back to her father and her father's family. Both girls enjoyed their childhoods, but they were so far apart that they really ended up more like cousins than siblings, and the mother only has a good relationship with the one who stayed in the UK. In fact, she has now (in her seventies) moved back to India to try to get to know her grandchildren.
The implications of choices made now can last a very long time.
Parenting is a hard old game, and I have no doubt single parenting is harder. I wouldn't say YABU to go, life is never that simple. But I think you do need to be brutally honest with yourself and assess how it will work straight away, in 5 years, in 10 years, etc. And put yourself in your ex's shoes and think about how it will be for him, and what you would want if the roles were reversed.
Good luck.