So I am currently going through exactly this. I stopped the contact and we are currently going through court. Do you think he's likely to go to court to fight for access and do you think your children are old enough/strong enough to tell cafcass that they really don't want to see him.
If they are firm in telling cafcass in their own words, not yours, then they shouldn't have to see him. But be warned this will be a long, awful process. And depends on his personality how far he pushes it with reapplying to court etc.
I have been told by cafcass, social services and police that no one can force my child to go kicking and screaming to see their dad, even if direct contact is ordered. If I had known this year's ago, I'd have stopped contact a lot sooner.
This is one of my biggest regrets, that I allowed him to continue seeing and damaging my child. I also wish I'd never put him on the birth certificate, as then he'd have less rights.
Ultimately, you have to trust your gut and do what's right for your children. But be aware that if he has parental responsibility and goes to court and the children say yes, they want to see him, there's not a lot you can do. Unless you can prove he's a significant risk to them.
If they say no, you have so much more chance. But it'll be an upward battle, as unfortunately the courts view is 2 parents are better than 1, even if 1 is damaging the kids. 😭
With your autistic child, might be an idea if they draw how they feel, if unable to say. Ask for a specially trained cafcass officer to help. Good luck x