OP you are getting a hard time on here and I can see both points of view.
The main thing is you need to strike up another conversation about how him attending Church is making you feel.
But, I want to pick up on this "family time" obsession that a lot of mumsnet seem to have.
I'm all for spending time with my family over the weekend - we all work hard and to spend some quality time together is great. But you know what? Spending time on your own is also great and very much required.
For example, in our family, DH likes his Sunday morning football on - he rarely watches TV and that's his thing he likes to do. We also organise our time to do stuff individually eg I might go to the beauticians, to gym, of run a few errands without DC; DH might meet a friend for coffee, go for a run, do some work; and DC havd there individual time too - homework, or reading, film, get the Lego out for an hour.
My point is that family time isn't always about doing stuff together. The weekend time is precious but it's finding the balance that you're all happy with: you time, his time, both of you time and family time.
The other point I wanted to make was that his attendance at Church will likely be more of a big deal than potentially he is making out to you.
I am a Christian, my Dh is not. I don't rub it in his face that I am but I also don't hide it. I don't put pressure on myself to go to Church every week, but when I don't go, I can tell the difference in my own health and wellbeing. At the same time, I do appreciate it is half a day of the weekend, so I tie that in with taking DC with me, whether they want to go or not. There's plenty for them to do whilst they are there and our Church isn't strict with what you can / can't do, so it the iPad comes out, then so be it. Because the flip side is that Dh gets Sunday morning to do his things (plus errands / housework etc) without DC about. My point here is that my faith and belief in God is at the centre of what I do each day and each week. Living with an athiest / no believer is difficult - your view points differ and he will potentially reluctant to tell you about it.
You need to find a balance; he needs to see that you're more getting "me" or "couple" time and you need to appreciate he has a faith that he is invested in and wishes to continue to nurture.
Some of the generalised comments on here about Christianity are unfortunate. As others have said, it's a shame we don't have the same tolerance for Christianity as we would other faiths. Sunday is the Sabbath day for me, we don't do a fat lot in general on Sunday's, and that's because we rest from the hectic 6 days we've had leading up to it! Despite living with a non believer, DH quite likes the 'do nothing' approach we've adopted - which normally involves some sort of walk after church, lazy dinner, and a film for us all to watch.
Hope it works out for you OP.