Oh Chuckle that's f*ing outrageous. Brilliant that you texted his mum, that you asked him to leave and that you threatened to call the police.
As for you "lying" about him coming into the house. Er, just no. If somebody deliberately insinuates something, you have good reason to believe it, then you find out it was otherwise, you were mislead. Ignore.
How ridiculous that you have no way of contacting him. What if something happens when the dc are with him?
You've made the first bold step. I suspect he was always going to sabotage it, to show who's boss. How about something along the following lines to his mum:
Following my earlier text to you, it is clear that x is unable to respect my wishes as regards drop offs. From this point forward he can accompany them to the front of the house. Once he sees that I have opened the door he is to leave. It may be our shared property, but it is now my home with the children. I have the right to privacy and to live here without the threat of x entering and behaving inappropriately to me, and in front of the children.
I will not be discussing anything with him face to face. If he needs something from the house, or there are matters affecting the children, he will need to communicate with me via yourself.