totally agree with @PrizedPickledPopcorn here.
He will absolutely take advantage of any hint of flexibility... "I thought you wanted to be flexible, so why can't I drop them back to you at 7.00 am on Friday"
It's a total pain, because in a better universe it would be great to have flexibility. But for him it will only ever work in his favour.
I think you have to be very economical with words and be very specific in any dealings.
I totally get why you don't feel up to a court battle. After All its only been a few months - but in the future, I think it could be quite empowering as you get to have your say and others will be examining and judging his behaviour.
But at the moment, I think you are wise to just get through it day by day. You are doing so well and I hope you have a bit of a break and some space to relax or distract yourself this coming week x