He can't actually stop you unless he goes to court, but you might be facing a tough time, either immediately, or long term - legally, your children have a right to a relationship with BOTH their parents, and BOTH parents have a responsibility to ensure that they can.
Can he offer the children a stable home? Is he in regular contact with them now? If so, he could apply for a specific issue order, or even an emergency residency order, to prevent the children moving with you.
Alternatively, you could move to Brighton with them, and then face him going to court later, where it is likely that a court order will be made about contact with him. If you lived close by, then the contact could be every other weekend, or one/two nights a week - but if you are a long way apart, then the court will have to order contact for longer periods of time top make it practical - so if you move, your DC's might spend a week or so with him at a time as they get older; perhaps it would be better to stay close if you can?