Some women would use that as a reason/excuse for they couldn't have done it though even when they have. This is why male abuse is still considered a joke to a lot of people which is totally wrong. It is just as serious as any other abuse.
I am 5ft 4, my ex was 6ft 2/3 it would have taken a fair bit for him to react to anything which would have meant I could and I honestly know that I could have done him some really serious harm if I wanted to or was that was inclined.
The thing is despite him being my ex neither of us went around getting drunk, argumentative and physical we respected each other.
If you are going to make this relationship work then you need a serious sit down and talk.
Clear the air over his texting. You cannot clear the air, forgive and then bring it up whenever you are pissed, annoyed or anything else. It has to be laid to rest or you will never move on. If you cannot get over it perhaps the relationship is not going to work.
If this is the first time that you have been physical with each other then you need to make it clear to each other that is is unacceptable either way and if either of you do it again the relationship needs to end.
You are saying that you couldn't have hurt him, he is saying you started it by being physical first. You had both drunk and you were angry.
You actually sound like you just need your heads banging together and to be told to behave.
As I said before good luck and the pair of you stop drinking so much that you behave stupidly.