I just wanted to say I'm so sorry for the situation you're in.
I suggest that while he thinks it's okay to harass you via text messages, you block his number and his family members and do not answer the phone to unknown numbers.
Get a cheap phone and let him know that number. You can switch it off regularly.
His mother is nothing more than a fool, and you should have told her to leave your home. How dare she come and talk to you like that. With a mother like that I'm not surprised he turned out to be what he is.
Quite frankly, if all he can do is be hostile towards you, I'd cease contact and he can go through the Court for access.
He can pay for the mediation and having a bully like him, I'm not sure mediation is the way forward.
Please keep a log of the abusive messages.
Keep a log of any other threats they make.
As you feel they are likely to want to take your son, I'd insist on supervised visitation or none.
Lets see if he will bother with court, as he won't even pay for mediation.
Unless the abusive messages stopped, none of them would be able to see him.
If you feel threatened, make a police report.
Is he on the birth certificate?
I hope your son has your surname too.
Please don't let them bully you.