Newmumsuchfun,
you should report that incident to the police. Does your neck or back hurt?
Do you have any bruises?
He shouldn't have restrained you to start with, or block you from going where you wanted.
My ex also accused me of hurting him because I scratched him on his wrists to make him release me. Tough.
It's called defense, not aggression.
But, if you report this, you can get him out of the house. The police should take him for questioning, and you can start legal procedures to ensure he doesn't have access to you or your baby.
In any case, you can always get to a refuge. Contact WA if you have nowhere to go and for general advice.
It may seem an exaggeration what I'm saying, but you are vulnerable with a baby, and once they start, they will do it again.