I do see OP's point that a big man hitting a small woman is potentially more dangerous/risky than the reverse - it's part of the argument against hitting children as well. But I also think it's irrelevant here.
I sometimes play-punch DH in the arm or chest in a "you are so infuriating" kind of way but since he is built like the proverbial shithouse a lot taller and broader than me, and I'm not actually putting any force into it, we both know it's a joke. He might play-retaliate by catching my wrists as per pp but as I say it's a game and we both know it is and each of us would be mortified if the other were actually hurt.
What concerns me about the situation in the OP isn't the physical actions exactly, but the reactions and thought processes - "disrespect", "you made me do it" and so on, on both sides. I would argue that those are the red flags and the disturbing/damaging things, rather than being physical with each other.
So, bearing in mind that we are a more physical couple than many on this thread, I still wouldn't expect to slap DH in an argument, and if I did I wouldn't expect to be shoved hard enough to fall down. And if either of those things did happen, we would be apologising and seeking help, not blaming each other.
Hope you can work things out, OP.