Depends on the friend really.
I have two small kids (one toddler, one newborn) and I'm one of the first in my "circle" to have children. Some of my friends are fab with kids - they come over at reasonable times that work around the kids' bedtimes, they give the toddler lots of attention, they don't expect much, and (for me, most importantly) they don't stay too long! They're understanding when you have to cancel. I guess they just "get" that children are pretty all-encompassing and that life does need to revolve around them, to a large extent.
I have childless friends who are completely and utterly clueless about the practicalities of having kids. So they'll turn up at the door at 6.30 expecting to be fed, they'll stay till 11pm, toddler will refuse to go to sleep for fear of missing out etc. I can't properly chat to them because the kids are demanding attention and I can sense they're a bit eye-rolly about it.
When the latter want to come to visit, yea, I do dread it. The same friends have no concept that I can't do a big, heavy night out on a whim. That sometimes your child getting sick means you need to cancel your plans (yeah, when DD has a temperature of 41 and needs to go to out of hours, I'm not going to go to the pub with you instead). That sometimes childcare falls through. They are impatient and intolerant of this, and as a result I get the dread in my stomach when I need to cancel or rearrange things.
They cause me stress because their attitude towards it all is pretty shit. So yeah, I find myself drifting away from them.