"I’d rather know what I’ve done wrong, instead of just guessing."
Why are you so sure you did something wrong?
"I did ask him and he just shrugged it off with, ‘I don’t want to start an argument‘. "
By doing that, he keeps you on your toes doesn't he? You are now trying to work out how you made him angry, trying to make things better, whilst he sits back and has you dance to his tune. You are taking all the responsibility onto your own shoulders for his anger. No - the responsibility is all his.
My gut tells me you did nothing to make him angry, but he somehow gets pleasure from making you "confused/angry/sad". If he were any sort of a human, he'd tell you what was going through his mind that made him angry. He'd apologise for making you sad. He'd make an effort not to be angry in future. He's doing none of this, just leaving you hanging while he plays the puppeteer and you wait for his next unfathomable outburst.
He's a wanker.