I don't think YANBU at all OP.
Firstly because of the limited family time. You say that you have jointly decided that the DC (and you?) do no regular activities at the weekend (you are also unable to commit to any during the week). Yet he does have a regular 'me time' activity at the weekend.
I think you need to discuss that again, perhaps considering the idea that any or all of you could commit to regular Sunday morning activities. The problem will be that you end up doing all the ferrying about for DC activities, so they'd need to do something compatible and you may find yourself sitting beside a football field instead of at home, which might not seem an improvement.
Secondly because, to quote a poster on p.1 ' People need time and space in their lives when they aren't just parents or spouse'. You need this too. Where's your half day 'me time' activity? Your opportunity for contemplation, community-feeling and spiritual renewal? How you achieve that is up to you of course.
You can't do this during the week. You could, as he does, demand your own half day every weekend that he's home, couldn't you. (Of course he gets this and presumably more time off to himself while he's working away too. In fact he must get a huge amount of time - probably more than he wants - when he isn't 'just a spouse or a parent'. You don't, so perhaps you're owed more when he's here). You'd really need to make yours Sunday afternoon or evening though, so there's still a full day left for any day trips as a family. Does your preferred activity run / can it be done at that time?
But, if you take half a day each and perhaps spend each others' half facilitating activities for the DC, that leaves only one day of the weekend all together. That isn't much if it's only on a handful of weekends a year.
I think that getting time to oneself and an opportunity for contemplation, prayer, reading, whatever one wishes, is important but, that can be done alone, flexibly, at times that fit in around family life. Yes, the sense of community that church attendance provides is different from this and enjoyable but it's not essential to maintaining faith to attend every week (for most churches, most people).
I do think that declaring half a day of each weekend as his own 'me time', without discussion, is selfish and inconsiderate. Just the same as it would be if his chosen activity was a sport, pub-going, or anything else.
I also think you should look into getting a regular babysitter, so you can go out one evening a week, every week.