Oh crikey, pinkchampagne, I have no real idea of what went on before this so am coming it at kind of blind, I think I managed to get the gist through other posts though.
This man has no control over you, and no say in what you do.
I split with my DC's father over 2 years ago and he was such a fucking wanker tool. I had the things coming back through the children, questions about what I was doing and with who, accusations of being in places I wasnt, whilst at the same time hearing all these stories about the girls he was sleeping with and even had some deluded young girl (practically a child) coming into my work telling me to 'leave him alone, he finished with you and doesnt want you back, let us just get on with it' (I ended it and was getting begging calls at least once a week).
I even had him call me one morning and ask why I was awake at 4am the night before, apparently he had someone drive past my house after they had been out and saw my light on!!!
Until you make a stand and put him in his place this will carry on Once he knew for definite that I was seeing someone he would drive past my house to look for their car, and was banging on my door at midnight, pissed up with friends saying he ahd come to 'join the party' thinking the guy I was seeing would be there. This is not something you want to happen so you must take action.
You need to talk to both your children and your parents to minimalise any damage he ma try and do. Obviously you don't have to explain yourself to them, just drop things into conversation. Do you have many male friends?? I found it easy to spend time with my DC's around a couple of my male friends, sometimes with their girlfriends too (if exdp had told them they were my boyfriend, dd would have set him straight 'no, he has a girlfriend, I met her shes called blah blah'), so they got used to the idea of mummy having men as friends.
Anticipate everything and act before he does, IME they can become very snide and seem to think that as you are the mother of their children that you should live your life as they want you to, whilst conveniently forgetting they are doing whatever they feel like!!!
I really hope you manage to get this sorted out, and I agree with everyone esle, at the very least change your locks.