OP I feel for you and so sorry you are going through this. I know this dynamic well. And it is nothing less than insiduous gaslighting.
My ex, (DD now sixteen) does this with just about everything and has done over a number of years. I think some people underestimate how upsetting it can be and I really feel for you. Dealing with a narcissist is not the same as dealing with a normal, thinking person and some people don't realise this.
You sound like an organised, caring mother who is trying to do her absolute best for her kids. He is quite clearly pretending to forget arrangements, times etc and I recognise this as attempted coercive control. They do it in such a way that to others it can appear trivial. And such people are very good at playing 'Mr. Reasonable' in front of others. But you will hopefully find that courts and judges have seen this kind of behaviour before, which is some comfort perhaps.
After having dealt with this for the past eight years and still dealing with it as needing to deal with the last throes of a divorce settlement during which ex violated yet another court order, I want to say a few things that I wish someone had said to me years ago, things that ay have helped me.
OP I realise you have said you don't have a third party to do the handovers.
In your position, I would at present stop facilitating any contact at all and use the time and energy to be with your kids, do nice things if you can, and prepare for going to court.
If you use a solicitor or are able to, ask them to make it clear to the judge that what you want is stability for your kids and an organised arrangement. Make it clear that you feel that your ex is likely to keep coming back to court if you don't have this. Courts don't like people to do this as ultimately it is a waste of public money.
Before you get to court, try to work out what kind of an arrangment would work for you. Write it down and be ready to suggest it to the judge to go into a contact order.
Be aware that you will need to put things in place to support yourself emotionally in this process.
Someone on here suggested 'The Nurturing Coach'. Website. This is run by a former social worker and is all about dealing with a narcissist ex partner. Can't remember her name but she has videos about dealing with the family court and also offers one to one sessions which may be useful.
As others have suggested, going forward you are going to need practical measures to keep your personal boundaries around yourself and your kids.
Some things have been suggested here such as being very careful about communication.
Listing handover times and places in court order. You may wish to even think about a public place for example a public library or a train station, even the foyer of a supermarket where there is CCTV. So that there is no scope for ex to mess about and where it will be noted if he is late or doesn't turn up at all.
If you possibly can i would also get yourself a counsellor who is aware of these dynamics.
I don't envy you but you will get stronger as others have said. Unfortunately I have experience of all this, so I really don't mind if you need to PM me going forward and I will try my best to pass on something which might be useful.
Once again. so sorry you are experiencing this, stay strong.