Keep all texts, if they are Whatsapp'ed, download and keep them so he does not delete them.
Send him a letter by recorded delivery and keep a copy and download a copy of the post office POD.
In the letter, tell him you want to get things back on an equal footing for your child.
Explain that when you were pregnant, he told people the child was not his.
You both work fulltime, the child is in nursery during the week so you can work and you live some 2hrs apart. You remain where you have always lived.
Wanting the best for your child, it was reasonably agreed between you that he would see his child every other weekend.
He keeps wanting to change his weekends and on the whole you have been very accommodating (site various examples). However what he cannot do is contact you and demand the child on a weekend, that in effect means he has had the child three weekends in a row. It is not fair, its not what was agreed but most importantly it is unsettling for the child. Tell him you also have arrangements/commitments/activities planned for quality time together for you and your child on a weekend.
When he told you he wanted the child for the third weekend in a row, you firmly said no and he still had a family member turn up to collect the child and the confrontation is not acceptable for you or your child and as such, you ensured you were out. You were clear that the child was staying with you that weekend and if he still chose to send someone to collect the child, that was on him. Him threatening to call the Police because of his unreasonable demands is just aggravating the situation.
In essence, as a parent you want the best for your child, and you appreciate that on occasion there maybe special events where weekends need to be swopped by reasonable prior and MUTUAL arrangement but the share of time needs to be equitable and fair. Demanding the child for three weekends in a row is not fair or equitable.
In future, any amendments to the arrangements must be reasonably in advance and by mutual agreement but more than anything FAIR.
I want you to have a relationship with your child and this is why I have offered the arrangement. Whilst we don't have to get on, we do need to maintain a reasonable relationship when it comes to the well being of our child.
So I propose we re-set the relationship and start over. I re-iterate our previous agreement that you have our child every other weekend, I will only hand them over to you or a close family member and I am always willing to consider swopping weekends in expectational circumstances with reasonable prior notice and prior agreement.
Send this letter and then get yourself a zoom doorbell (do not point it out to them but don't hide it either) and record any exchanges on the doorstep and keep them, even if you don't think there is anything of note. You have already said that when you have been fine, he claimed you have been nasty, so learn the lesson.
Keep all text messages.
Then see how he reacts, this letter is very reasonable, yu are fair, yet stating the issues you are having. Good Luck