Falling out is more peaceful sometimes. You can’t live your life this way.
I agree, but it sounds like the only way that would be certain to happen is if they divorced. Even then, she could still persist and say "Don't think you can keep me away from seeing my grandchildren".
If it did ever go as far as divorce, OP would be thoroughly made out to be the bad guy and the one to wreck their children's home life. if your husband once punches or rapes you, or has a one-time affair, no reasonable person would blame you for divorcing him.
However, it's another matter entirely when it comes to behaviours that are perfectly acceptable - pleasant, indeed - on a normal frequency (and when wanted by both parties), but it's often difficult to communicate to others when normal behaviour becomes stifling and controlling, and you can often end up doubting yourself.
"Are you really saying you want to divorce him just because his mum wants to treat you like her own daughter and to spend time with her grandchildren?!"
"Hmm, I don't know - am I? Is that pathetic of me? Am I the controlling one after all?!"