Maybe, but other religious icons and symbols - usually Christian ( as that often seems to be seen as fair game) are often used as fashion items. Crosses, crucifixes etc.
I understand what you are getting at.......but fashion doesn't really care about those sorts of sentiments...more about iconography and symbolism/imagery.
Yeah - I don't think people should do that. It has nothing to do with cultural appropriation, it's just disrespectful of others to do it for frivolous reasons, be they members of your own group or some other group. And outside of woke nonsense, most cultures only have a limited number of things that are set apart in that way, so it's not a terrible pain in the ass to avoid it.
It's not like its even a free for all, anyway. If you wear medals that aren't your own, or the regalia of some office that doesn't belong to you, there are instances where it's illegal, a sort of impersonation. Even as a secular society we accept some things should be used in a particular way - for us the ones we protect legally tend to be commercial or privately owned, which probably just reflects the things we think are important.
One off the things that pisses me off about the idea of cultural appropriation is it often means people feel like any complaint about people using symbols etc is just the same stupid shit, and should be ignored.