I'm not sure I ever gave two hoots what anyone what thought personally, but I think most people do feel more confident in time.
A sling can be used in any temperature as long as it's appropriate (some are for warmer weather) and baby is dressed appropriately. I've baby (and toddler, and preschooler) worn in 30+ heat many times, often using a mesh wrap with the child mostly naked, and caring a sub n umbrella!
Dealing with a small child in public though I think gets harder as they get older. Carrying a 2yo whilst pushing a pram is a lot harder. Whilst I didn't care less about other people when my newborn screamed in public, I still cringe thinking about carrying a kicking, screaming toddler that's as stiff as an iron board. Or one that is screeching for chocolate whilst lying down screaming in a shop, for example.
I'd recommend trying to adopt a 'just let it all easy over me' attitude, if you can. Honestly, no one cares that your baby is caring. No one is judging you for it, and those that notice will have forgotten a moment later. If baby is other to crying be mindful of others where possibly - no normal cinema, theatre, no quiet zone on trains, but otherwise crack on with life.