Honestly I wouldn’t change the time.
The reason is that you said this: ”The only time I’ll ever need him to have them later than 7am realistically is if I do a Thursday night shift.”
That means that any time you do need him to have them past 7am, he’ll kick off and make it difficult. And it will become him “doing you a favour”….if he agrees.
You said that the reason for the later drop off is do he can take two of them to school. Perhaps In September when school is back, he won’t be reneging on the agreement - it’s just in the holidays when there’s no school, he’s in a rush to get rid of them.
If you were feeling kind, you could just accept them back at whatever time they arrive, without insisting he waits until 10am.
If you aren’t feeling charitable, I’d make him wait until 10am every time.
I think because you’re currently trying to establish boundaries and because you might sometimes need that later drop off, I’d be inclined to stick to 10am. Nice and clear, no exceptions, everyone knows where they stand. The moment you need to start saying “well it will need to be later than 7am this week,” the more trouble you court for yourself.
It’s not about punishing him, if you didn’t need the flexibility of a later drop off for some weeks I’d say let it go. But you can’t rely on him to be fair or reasonable so it’s better you take the necessary steps to make sure the arrangements work for you.