He sounds as though he bordering on stalking you, and taking an unnecessary amount of interest in things which are nothing to do with him.
My sister had to endure two years of this before she finally resorted to the law. I recommend that you do not wait this long- contact your local police force if you are concerned about his behaviour and ask to speak to the domestic violence unit- they should be able to advise you. This is particulalrly relevant if he is follwoing you or having you followed, which amounts to harassment and is in itself a prosecutable criminal offense.
This may seem extreme, but there are enough cases every year of former partners turning nasty to justify them taking it very seriously. Remember, you did NOT bring this on yourself- it is not your fault he does not know how behave in a sane manner. What you are able to do about it is not accept/ condone/ reinforce this behaviour.
To avoid reinforcing it, you must treat him like a toddler and try to work out what he getting from the behaviour, and then work on NOT giving him his reward. He may kick up a fuss at first and insist, but do not give in. If he gets his kicks from quizzing you about your life and making you squirm, refuse to answer any questions. Just say calmly that it is none of his business- do not respond in any way, even negatively.
He has not business coming into your house. Do not let him in if you do not want to. As someone else has said, you could arrange for the children to be picked up from a third party's house.
I really feel for you- do not think that you caused this- that is what he would have you believe, but it is not true.
One last thing: if you have any reason to suspect that he might become violent towards you or the children, do not do any of the above, but seek help and advice from the domestic violence unit first!!!