What are we going to talk about? And that's if they can talk! The stranger children approaching me in the coffee shop, what am I supposed to say? "Hello, terrible weather we're having isn't it?" That would be my normal approach to a stranger but doing that to a child would result in them staring blankly at me.
See, that really doesn't match my experiences, either as a parent or as someone who has worked with children. If I opened up with a comment about the weather to someone, I would expect a fairly lightweight, identi-conversation which would be forgotten as I was involved in it.
If you have a conversation with a child, you never know what you might end up talking about. I was recently schooled about sea creatures by a four year old who loves Octonauts - she opened the conversation with 'DoJo, did you know that sperm whales eat giant squid' and within 5 minutes we were online checking whether a giant squid's eye is bigger than a bowling ball (indeed they can be, as we discovered).
That is the kind of conversation I would find interesting from anyone of any age, and actually most kids have an interest in something that is likely to be of interest to adults. They might not have the level of knowledge that an adult has (although quite often they will pick up facts like sponges and suprise you) but their level of curiosity, the way they think about things and the questions they ask are just as likely to produce stimulating conversation as an adult in my experience.
It might be harder to get a conversation started with a child you don't know, but only because the 'accepted topics of conversation' such as the weather etc don't really work with a small child. But that's true of a number of adults as well - if you try and talk to someone about football and it turns out they have no interest, you try and find some other commin ground, not write them off as boring or impossible to talk to.
Whilst I agree that holding someone else's baby is unecessary and you shouldn't be made to feel bad about not wanting to, I think that dismissing all children based on the fact that you don't know how to start a conversation with one is a bit of a shame.
Fortunately for you, I can't imagine most children care one way or another (not being rude about you - as much as they aren't tuned to social niceties such as discussing the weather, they also usually don't give a shit whether an adult finds them interesting or not and will find something else to do if you ignore them), but you have to accept that expressing such thoughts to their parents will be taken as an insult.