Do whatever gets you through the holidays and then go to court!
If he is basically NFA then he can't have kids overnight. Try writing him a letter, well worded and straightforward. I say a letter, not text, email or phone call so that he cannot later twist your words to suit his own ends. No threats, no emotional outbursts just that:
As mum, not married to dad, you are legally the default parent with sole parental responsibility. If he can't make and keep to reasonable agreements, then until he takes you to court he won't have any contact at all. That decision is your right to take and, if he continues to upset both you and/or DS, then you will have to take it.
Basically, he plays nice or you take the game away. Remind him that it is possible to have lots of contact if he stops making life unpleasant for you, if he puts his love for DH first then you can all get along very well.
Children Act form C1 is on the HM Courts and Tribunal Service website should he wish to make contact formal. You could use it too! There is lots of free advice out there that will allow you to take this through court yourself - may even be a forum here??
Keep a copy of any letter for SS, maybe send a copy to the last contact you had and ask them to place it on file, in case ex reports you again.
But, whatever you do, do NOT let this man call the shots forever. Foot down, firmly.
Good luck.