Do you feel the same about people having phones with lithium batteries or electric cars, or wearing cheap clothes? (Absolutely not trying to be an arse here, or start a bunfight!).
High end gemstones of course have links to criminal activity, but it’s possible to avoid these by buying from ethical sources, or by avoiding precious stones altogether
Diamonds are a rather common gemstone, whose value is directly linked to criminal activity, money laundering, mafia type set ups. Other precious gems have similar links. It’s fairly easy to bypass this by not buying stones commonly sought for high end jewellery making.
I have a few crystals (mostly rough) whose provenance I’m not 100% sure of, but most of mine are bought via rock hunters funding their hobby, through social media.
My favourite rock is a large (but poor quality) carnelian which was picked up by a shop owner/rock hunter who was trekking in Oregon and literally stumbled over it in a dry river bed, and brought it back over to the UK. The same person has also personally found Cornish amethyst, fossilised shark teeth, quartz, herkimer diamonds, mookaite and more during his travels, very ethically and above board, and I’m happy to buy from him.