I agree, it sounds like he is 'playing games' of a nasty kind here.
I had a boss like this. He had a wife at home to look after the kids, whereas DH & I muddled through with nursery and after school clubs.
I was convinced he would try to 'test' me by coming up with ridiculous 'breakfast meetings' at 7.30 in the morning, and last minute 'rush jobs'. I had asked for a 4 day week and had been refused .
After a restructure I was made redundant, and his comment to me was, "I thought you'd be pleased - after all, you wanted to spend more time with your family didn't you?"
I should have sued him/the company, but of course most of the time people just don't have the energy/ inclination.
Defintely say something - keep it short and firm - "sorry, that time isn't convenient for me - how about one of these alternatives?"