"You're responsible for this situation...You're a parent now. Time to grow up."
I think people have been incredibly harsh on this thread.
OP - this may surprise you but I have been in a very similar situation.
I think the advice you have been given on here may be unhelpful. IF you think it is too hard to send that sort of confrontational message - so difficult that you can't actually bring yourself to do it - then it's bad advice. You may find yourself in a situation where you just end up carrying on seeing this person because the thought of sending that message is just too difficult.
However, if you are brave enough to do so - and I hope you are - carry on.
For my part, I sought similar advice from friends and was getting the same sorts of responses. Until one person, who I love and respect, simply said "fuck all that. If you don't want to write to him, don't. You don't owe him anything."
Key difference - he doesn't know where I live.
I hope you find the strength to do this, OP. Or get your husband to do it for you, even if it's just him hitting the "send" button. Too often we become isolated with these problems. That's how these men insert themselves where they are not wanted.
In my case, the last time I saw my stepfather, I had to ask my DH not to ever leave us alone together. Even with that warning, we were in a group and my husband drifted off. In those few moments, SF managed to "get me alone" and start talking about a visit. It's fucking awful because years of socialisation stop you from shouting out "DH, I need you over here NOW."
To all the "tough love" posters on here - you don't know what it's like until you've been there. Really.