'As I made him so angry'
Right so, if he hit you would that be because you 'made him so angry' too?
What you are talking about is abuse op. He didn't punch the door because he was angry, he punched the door to scare and intimidate you.
That's what abusers do. It's to say to you 'if you don't behave the way I want, this could happen to you'. It makes you shut up. It makes you walk on eggshells, worrying you might do something to anger him.
You're not to blame for him punching walls. Punching walls is what sociopaths do. It's not normal. And based on your update he's gone from crappy father to abusive garage.
Get out of there fast.
Don't raise a child in a household seeing its mother abused by its father. Its not ok.
There's no excuse for his violence. There's no excuse to damage your home or scare you and the child.
Get out and never go back. If he wants to see the child, always do any handovers and drop offs in a public place.
This is very serious op. Speak with women's aid. Get out. Do the freedom programme online.
And above all, remember this- you do not 'make' anyone violent. He is violent because he chooses to be.