How old are your children? I take my 2 to church on my own as my DH isn't a Christian. It took a huge effort of willpower at first, because I had been hurt by the church I'd be brought up in.
Now, I feel I've really missed out if I don't go! Not that its always easy, as DS (6) has Asperger's, and his behaviour isn't always as good as it could be.
My opinion is that you CAN be a Christian if you don't go to church, but it is really hard, and you could really do with that fellowship that comes from being with other Christians. More importantly, your children would benefit too.
As for broaching it with your DH, you might be surprised! Mine isn't a Christian, as I said, and I was a bit concerned how he'd react to it all, but he's been fine, even comes along occasionally!
Sounds to me you're being prompted to go, but are putting obstacles in the way. As someone else said, why not make a commitment to attend, say once a month and see where it goes from there?