It can be really difficult trying to entertain and control a group of small children and make sure your childless friend isn't picking at the hem of her top, flicking through her phone or staring at the clock through sheer boredom, or rolling her eyes in frustration at the conversation being interrupted again by a child needing a wee.
But this isn't how I feel and my friends know this. I spend time with their kids and play with them as well as chatting with the parents.
Not all childless people are like what you have described above.
By not asking, you're not giving the childless person a chance to spend time with you. You're alienating them by being exclusive.
Just because you couldn't imagine spending time with other people's kids doesn't mean that someone who doesn't have the constant pressure of parenthood wouldn't want to. Especially if that's the only chance for them to actually spend time with you.
You don't have to entertain your childless friend or make them happy, they would be there because they want to. You're making out that a childless friend is as much as a burden as your kids. Which I guess is the point of the OP. She's not special enough to join the club. That's hurtful.
Keep on excluding them and eventually you'll lose them. They can't make an effort to come see you and your kids if you don't invite them, which really shows your lack of interest in the friendship.