Sorry to read your having a stressful time. Me and DD dad split prior to me finding out I was pregnant, so I am a single mum too. Although we are NC, so I don’t have the stress of dealing with him. I do have some experience though from my ex SDC.
What are his demands? 50/50?
Are you claiming Child maintenance? If not, do so, and with the way he is go through CMS.
Keep a log of every bit of communication in a note book, every late drop off / pick up / no show / threat. No matter how minor you think it is.
Is he on the birth certificate?
Has the mediator contacted him directly to attend mediation? If not, ask them too. This way it’s done by an independent third party.
If he has refused mediation, they (mediator) should advise you to instruct a solicitor to do a court order. This can be expensive, so I would check if your entitled to legal aid. It can also be time consuming.
It is best if you can mediate and agree on the terms of childcare, but does seem like your ex might keep moving the goal posts, and not agree. So I would say court order would be the better option, and he would have to stick to it. Although you do find they try and make it equal as possible, unless there are concerns over welfare.
My ex had 3DC, and their mum refused mediated or to agree to anything. So we had a court order, it was 50/50 but as she didn’t stick to it, and we had concerns over the DC welfare, we got full custody in the end with supervised visits. It was stressful, and at times we just thought we should stop fighting, but although it did take time, it was worth if for the DC in the end.
How old is your child? I would say to try keep all communication in written form, so a text message “what time are you dropping DC off?” Reply 5pm. Then if it’s at 8pm with no communication you have evidence. Sorry your finding this so stressful, and my suggestions might not make it easier.