you are putting your self in danger.
you really need to report him to the police and get a restraining order, you need to see a solicitor and discuss PROPER|LY the issue of a violent man having access. there must be a law that accounts for this.
I think a good first step is going to the GP well done.
I also think that if you truly believe that a man who punches the mother of his children - is a good father , and that those two issues can be seperated, you are very, very mistaken
if he cared anything for the children at all, he wouldn't play power trip games and fuck yourhead up - not allowing you to function properly as their mother.
would he?
he doesn't care for their wellbeing very much at all really does he?
if you don't take the initiative of finding out where you stand legally with regards to allowing a violent man to be violent to you whilst the children are in the house, you may be setting your self up for a huge shit storm with social services.
so please, i beg you, see a solicitor. report him to the police. get a restraining order.