There is no set rule on distance.
It's about the impact on the children's relationship with the non resident parent.
So anything that makes school days difficult will be considered more carefully than if you're close enough to share drop off etc.
You don't want to end up in a situation where you are forced to give him more weekends and holidays to make up for you moving away.
Can you afford to stay put until the house is sold and see what he does then?
If he moves then it's much easier for you to do so too.
If he's not been interested in parenting at weekends when you were together, how likely is it that he will be happy to do it all weekend every weekend long term?
My sister refused 50/50 with her ex.
Three years of court proceedings, 13 hearings, 4 carcass reports, £40,000 in legal fees and zero possibility of her ex ever agreeing to do her a favor in the future. She ended up with (shift patterns make things a little complicated) 7 months of the year to his 5 months, almost 10 years later she accepts that those 30 days a year weren't really worth the price.
It's all very fresh and new , take a breath and give yourself time to decide if you really want this to be your future.
I'd start by asking your ex to suggest another date and only disagree if it's actually unworkable. Take it slowly and see where you are in a few months.