How far is "miles away"?
We moved miles away from my wife's ex... an entire 5 miles, with a direct train connection. He literally called it "miles away" and filed for a PSO - which as you might expect went basically nowhere. Different people have different ideas of "miles away" - if it's less than 1 hour travel time, that's not a big deal and has minimal impact to contact. If it's more than that, it would still be allowed, as long as it doesn't overly impact contact, but it'll mean more travel.
If you have no court order presently, you should just move. He may take you to court but it will likely go your way because:
- you informed him and made an attempt to work out an arrangement but he has made that difficult (but key point is you didn't just surprise him)
- there is no court order stopping you
- once you move (literally, within days) that will become the new status quo
He may or may not actually take you to court. If he does, you might get a bit of a talking to that in an ideal world you wouldn't have moved, but the reality would almost certainly be "well, you've moved, it wasn't illegal, and now that you have moved, we figure out the arrangements from this point forward".
You could try to pre-empt him with a Specific Issue Order granting you permission to move, but actually you have a higher chance of that failing than just doing it anyway.
In any event, you will most likely have to facilitate travel and take the responsibility for more than 50% of the travel, so keep that in mind.