So full disclosure, I'm a weed user, and I use it in small amounts to stem my anxiety and pain related to my physical, mental and neuro issues.
I don't like to be heavily stoned at all, it's unpleasant. I don't drink.
I believe certain types of drugs should be decriminalised, but the ones I think should be kept illegal, are coke and heroin. Both of them are godawful. The damage and violence born from them both is beyond description.
Saying that, you are clearly not at all compatible with this group of people, and truthfully their behaviour is so far part reasonable, that I would run a mile.
I know folks who partied with coke and heroin in their twenties and thirties, but then the ones that survived it, settled down and got on with their lives, at least one has his own family now.
My advice would be to just disengage, don't go into too much detail, but say that you're not comfortable being around drugs, and that leave it at that. If they get shitty then you KNOW you made the right decision.