This post makes me feel very anxious as it takes me back to my own divorce.
DS was 4. His father became involved with another woman very shortly after we split up. She didn't know me well, although she wasn't a stranger. She immediately set about destroying my reputation and trying to remove me from DS's life.
She arranged for 50/50 custody (as it was then) but it turned out that this meant DS was spending at lot of time with her very elderly parents. Her father, who was in his 80s, would collect DS from school and he would have tea at her parents' house. The father used smacking as discipline and told DS that he wasn't to talk about me as I didn't love him as much as OW and he would soon not be seeing me anymore.
OW would give instructions to school, HCPs and others, kept all his nice clothes (chosen and bought by me) and sent him home in clothes that weren't new, didn't fit and didn't belong to him.
It all came to a head when they got married and they tried to pretend they were taking DS on honeymoon with them (to the Maldives). In fact, they had arranged to leave him with XH's parents rather than return him to me. It turned out that they planned to use this as "evidence" that I didn't want him. I was having none of it, involved lawyers and went to collect him after the wedding as per the consequent court order.
When I got there, the XPiLs had disappeared, leaving a note stuck to the window telling me that I wasn't ever getting him back and I should leave them alone. This was reinforced by XBiL (who was at the house) telling me that by the time XH got back from honeymoon all the legalities would be in place.
Cutting this very long story short, it took 3 weeks, hot and cold running barristers, a court order and threats of involving the police before my child was returned to me. XH's contact was severely curtailed after that, but he carried on with legal action to get custody, eventually giving up about six months later. DS hasn't seen him for 20 years.
OP, cease contact immediately, don't worry about seeming unreasonable and do everything and anything you have to do to get back in control. I know two women who did not take advice, went along with the being reasonable scenario and both lost their children. Try to remember that while it may be very frightening and you will feel threatened and alone, there are no real, effective sanctions that can be applied to you if you refuse contact - your children are very small and the relationship with the mother is the primary one.
Your XH can huff and puff all he likes but he can't blow your house down.