Ex won't have kids when I'm at work,am I in the wrong?

[nating]

I work part time 20 hours.
My ex doesn't work and neither does he's gf (he has 1 son with her and her 2 kids )
He pays me no maintenance and has them Saturday night (3 a month as he wants 1sat off )
I've started a new job and I'm classed as a key worker so they will be going to school.
Half term I have to work (as a new starter at my job I can't put holidays in till the following half term)
I have no childcare for the (Monday ,Tuesday and Wednesday )
I asked him could he have them as I can't stay off work on my second week.
He said
No I have them Saturdays,I will have them the Monday but that's it,I'm not your private babysitter,do what others do and work around your children.
I can't have 5 kids in here,sorry not going to happen
Take it or leave it!!

This is a man who doesn't pay a penny,doesn't work etc etc
Im not wrong here am I ?

*How has he managed to not work for so long?

I thought the job centre were very strict these days about making people look for work.

Is he actually employable?

Maybe you should consider reporting him for benefit fraud.*

Maybe he claims pip etc? My ex hasn’t worked in at least 5 years because he is on pip so only pays £7