Sounds like you might have it too loose, and/or a different style of sling would suit you.
With a stretchy one, you need to first put it on without the baby in, tie so there's no slack, and then let the stretch of the material take the baby's weight rather than leaving room for them, or tying it around them (like you would with a woven sling) - the stretch will get looser as you walk, so ideally you need to start out with it a little bit too tight.
Also, preferably as few layers of clothing between you and the sling as possible, so have it under a coat definitely, maybe under a cardigan/zip top/fleece/whatever as well.
If you have a sling meet or sling library near you you could go and get some pointers and try a few out. I know sling meets get a bit of a hippyish/bad rep on here, but this is the exact situation they were invented for, and you wouldn't have to carry on going if you weren't keen - they're happy for people to stop in for advice and never come back 