I think the trouble has been that you and your DH have not been properly in this together. From the start, (if only he had realised) he should have included you in the parenting decisions/rules of the house. Not only included you, you should have both been the adults in charge together of this whole family.
Your adult relationship is the PRIMARY relationship in this scenario and should have been a massive consideration, in fact top of the list of things to look after. (The children ultimately benefit in a situation where the adults are mature enough to have a decent, respectful relationship)
Unfortunately, thus far this has not been the case. For the DCs to feel really happy and secure, they need to know that the primary relationship is intact, together, and in charge of the running of the household.
They needed to see their father and you showing respect for each other. They needed to see they could not play you off against each other to get their own way. That they could not get their DF to side with them against you. (Unhealthy for them to have that power) Then you and DH could have talked about what was best for the children, the household, yourselves and the general health/harmony/welfare of your family life.
Unfortunately, none of us gets a manual on how to be the best parent or spouse we can be. So your Dh was unaware. As their father he has probably done what he thinks is right. He has underestimated the need for you two to have been together and in charge of the family.
In a situation myself some years ago which had similarities to yours, I went to therapy and after two sessions, the therapist asked me, as this was a family issue, why my DH wasn't there at the therapy. I said that I had been the one that was unhappy understatement with the situation and not managing, so I had come for help. She suggested I ask him whether he would like to come to the next session.
I am very lucky that he did want to, and we learnt all that stuff (in my above paragraphs) and things really started to improve. I am very lucky that he was the kind of man who was able to accept he'd made mistakes, by trying to stop me have a say with things going on in my own household, by enabling DCs to play us off against each other. I am very lucky he wholeheartedly participated in the therapy and he actually said to the therapist when she'd pointed out where he was going wrong, with diagrams and everything "FantasticButtocks has been telling me this stuff for years and I haven't believed her." It was so good for us both to get help to manage a difficult situation that wasn't making anyone happy. We were also lucky we had a good therapist who knew her stuff (CBT). DH and I did about five or six sessions devoted to this subject, with homework, and the whole family in the end really benefited.
A few years later and all our DCs are grown up and leading their own lives and we are free to enjoy our relationship. Just told you all that OP, to say, it could^ work out if both of you want it to. You sound beyond stressed with it, and I know how that feels. Something needs to change.