Ex and looking after his dc's for a whole extra day

[Heywhatawobderfulkindofday]

I'm pretty new to this co-parenting (as of Jan this year) and need to know what's normal in terms of changing child care arrangements.
I have our two DC's five days a week, he has them for one weekend day and one after school. I drop them to him as he doesn't drive. This means if I go out on the weekend night he has them then I can't drink as I have to pick them up. He can't have them to sleep often as he lives in an overcrowded house.
I haven't ever mentioned formalising this arrangement although I think I should now. I was hoping we could keep it flexible in a way that would benefit us both.
I've asked him to have the dc's for three days in September (plenty of notice) so I can go away with a friend for a long weekend. He has said no. No reason but I expect it's a form of control, as he used to do this when we were together (refuse to look after children so I couldn't go out on the night without them).
I'm devastated and don't know what to do. The problem is that he doesn't socialise and if he does ever need childcare during his days then he has someone who will look after them so he can go to the gym or run to the shops. I don't have anyone who can do that so when they're with me, we're together all the time.
I was annoyed and I told him that as he said no then I would have to consider whether I carry on driving them to his every week but do I really want to be this petty? What if he just says fine and then never see's them? Then I'll have no childfree time.
I need advice as I can't imagine never having a weekend or a night out again. Why does he have to make things so difficult when the majority of the time it's me doing everything?

I can understand its annoying, but my first thought if he wasnt going to see the children would be for their welfare, not my own. If you want to stand up to him then your children will be potentially the ones who lose the most.
However I wouldn't drive them, Id say to him id like you to reconsider how we agree we will support occasional changes moving forward and if not then we will formalise the agreement and maintain your boundaries, its up to him to collect his children, having you running all over the place is also control.

@Kiera220 yes I did think of their welfare too. It wouldn't bother me that much and to be honest if they're sick, they stay home with me, or if they have a birthday party or if they need a quiet day at home. I can just see that he would slowly stop contact and then somehow blame me for making it 'difficult'.

You will be so much happier if you can arrange either paid or reciprocal friends’ care for your kids. Trying to make him be a responsible, involved parent is going to lead to a world of pain and will not work.

do you gave the CMS claim in yet? If not, put it in. Men like this tend to respond to ££ over moral imperatives. If he thinks he can deprive “you” —his kids— of money, he may agree to have them more. Sad but true.

@Radiodread it's a big ask though isn't it, for a friend, having kids for two days, overnight and then take them to school the next day.