OP, you say that he 'harassed my dad about contact with our son so I had to unblock to arrange contact before it worsens for my dad.' Why didn't your Dad just tell him to piss off? As I said to you before, unless contact is ordered by a Court, you don't have to give him access to your child. It doesn't matter what he says or to who, you MUST make yourself ignore it. If his name comes up when with your family, just tell them you want nothing more to do with him, you don't want to talk about him, and please change the subject. If he genuinely wants to spend time with your joint child, then let him take you to Court, but I bet he won't because he's not interested in your child for his own sake, he just wants to use him to manipulate YOU!!
You've let him pull you back in, yet again, ask yourself why?
Then come back and tell us why you think you did it, rather than just ignoring him.
If he's bothering your Dad, and your Dad wants it to stop, then I'm sure he's perfectly capable of telling him so. If he visits your family, it's up to them whether they choose to let him in. If he calls your family, it's up to them whether they choose to speak to him, but none of that is YOUR problem. Tell them that you want nothing more to do with him, you don't want to hear about him, and it's up to them whether they want any contact with him, but if they do, YOU don't want to hear about it.
You say that you've blocked him again, so is he currently in touch with the rest of your family? You seem to only tell us part of the story, but if you REALLY want out of this relationship, then don't drift away from MN, we can really help to support you, but you've really got to WANT to get away, and stay away from him, otherwise you're not moving on, you're just prolonging the agony, and not just for yourself, but for your joint child too!