I actually think it quite rude of the husband to re organise and suggest what time he wants dinner! you can't do that if you are invited to someone's house. You simply cannot dictate what happens at someone elses dinner party. I'm really shocked tbh.
Whilst I understand his reasons, it's just not what is going to happen. Simple. you need to be honest with him. Tell him you have given it a huge amount of thought, and that you are terribly sorry, but since you have invited others, and whilst a children and adult evening would e something you'd love to do, on this occasion you are sticking to your original plans. That is, to have an adults dinner party, at an adult time for dinner.
Also stress that the other couples invited have been looking forward, as have you, to an evening without the children to worry about, and you feel it unfair to change this for one couple and disappoint 3 other couples.
If they really won't leave the children, then perhaps the only compromise you should make is to say that they can bring their children, but dinner will be at the original time you'd said.
Suggest you will plan another evening soon, more to their agenda, and would look forward to it, and include everyones children.
SO difficult I know. However, you really need to stand your ground on this, as no-one will have a good evening, including you at this rate.