OP, imagine that you'd had the conversation with him, and he said that he'd have the kids full time, and you'd instead have to just see them in school holidays (realistically too far for a weekend). Would you have said ok, and then gone for the job?
I very much doubt you'd have been happy with that, because you value the relationship with your children too much to just see them occasionally. By the same notion, I think it would be unfair for you to move, and expect him to give up a day to day parenting role, to just be a holiday dad. Its unfair on him, but most of all, its unfair on your children. You'll be influencing the lifelong relationship they have with their father.
Its not about you having to go where he goes - neither of you have moved away, but its unfair that by you moving away, he has to either choose to follow you round the country, or only see his children occasionally (or have them full time).
Bear in mind as well, that given how involved he is, that he does all pick ups, works from home etc, that he may well change his mind about having the children full time. The realities of it may well not have hit, and when push comes to it, it wouldn't change his life that much to have them full time by the sounds of it, whereas you are in the realms of after school clubs, au-pairs etc.
I do feel for you, I really do, but sadly I think the children's relationship with their dad has to come first right now.