I think you know exactly what he’s done OP, I don’t why we feel the need for the truth or a confession...all it does is confirm what we actually already know and it bloody hurts too!
He absolutely does remember what happened, he remembers exactly, he’s just determined not to be honest and tell you about it. Why? Because you’ll misunderstand, take it the wrong way, think that it’s more than what it is? No, it’s because it’s exactly what you think and he knows this.
Last time he got off lightly, you didn’t push as much for the truth as you might have because at the time it felt like there was a lot at stake. Purchasing and moving into your own home, should be a wonderful and happy time right? Who wants to believe their partner cheated right before all this, when you think your both happy, so happy that you are investing in buying a home together. He knew you didn’t want to upset the apple cart too badly so he lied, minimised and deflected until you just shut up about it. Now that your in your new home, settled, having just been through lockdown together and he has falsely thought you were satisfied and the coast was clear for him to go right back to whatever it was he was up to. You’ve just shattered that notion for him and THAT is why he is angry. He is angry that you’ve just made whatever it is he is doing or about to do so much more difficult as he now knows he can’t just come home late/leave early or whatever it is without you being suspicious and without being questioned. He now has 2 choices he can go to a lot of effort to hide things, lie and put up with arguments when you don’t swallow his BS. Or he can just stop what he is doing. He clearly resents doing either, hence his anger at you.
I don’t know what kind of man he is OP, but is it possible that he thinks along the lines of he’s given you what you want e.g sorted out his money issues, bought a house with you, willing to have a child, and think you should be bloody ecstatic about it and therefore feels entitled to do what he wants and that you shut up and put up? Or maybe it’s that he thinks he has you well and truly trapped now with the house and everything and so you won’t leave?
I think you going away for a few days is a very good idea. Get some space, some distance and have some time to think. It sounds like he never grovelled and begged your forgiveness the first time round and simply placated you with a bunch of excuses and lies. The important thing here is that it doesn’t sound like he was sorry, he’s wasn’t sorry then and he is not sorry now, just angry!