I think you feel negative about going to court, but as other posters have said, it would probably be a better outcome for you than you are envisaging.
However, going to court is a big matter, so how about trying something else too. Go to Mencap or other similar family-supporting charity and ask if they can help you to secure the services of a carer to help with outings for your child who has SEN. They might help you to secure direct payments for this, or augmented PIP , or they might have a register of volunteers. Sometimes Mencap gain funding for projects which involve placing volunteers with individuals who need a little more help to access activities. Sometimes, there are students who wish to augment their knowledge and practical experience through volunteering. If you could arrange a situation where a carer does things with DC, then you might be able to persuade your Ex to have their help too, to help supervise and prevent wandering off. That individual might be able manage some of the communication with your ex , so that it becomes more matter of fact and less toxic. If you were to gain support from a social worker, then they could help to set up regular, safe contact and make the communication less tit-for-tat and about exerting pressure.
Christmas is something you can accept. It seems to be reasonable to share, even if it hasn't previously happened. You can have traditional celebrations in the days following, and do something lovely with the kids for New Year.
Presumably, it will be your turn the following year. I know where you are coming from, but it will seem unfair if you don't share.
Fwiw, I think that the notion of family members losing children who have SEN is a very real problem. I don't believe that everybody has that inbuilt capacity to be always anticipating and responding to problems - however much they care. Eternal vigilance is a difficult skill to master especially within environments which are not safeguarded. I haven't been mother to a child who does not have SEN, so I don't know, but I presume that kids without SEN tend to reference what their carers are doing/ expect them to be doing at any one time, and it's easier to keep them safe. Certainly, I would hope that a charity support worker or social worker might be able to help you to express your concerns in a constructive and positive way and to draw a plan to represent the risk assessment and to say what can be done to minimise wandering off. It would seem very unreasonable on his part were your ex to refuse to read such an assessment and to contribute to it.