I'm hoping that because access is regular and consistent that perhaps you may have decent channels of communication with your ex? If so couldn't you just express your concerns? It's not at all unreasonable to not want your little one around drug use, and is there any chance he might be able to see your point of view?
I'm kind of in a similar situation at the moment only I'm the bloke in the equation, and whilst my ex and I did not split on the best of terms I am very aware of how much a struggle it is for a Mum. In my case the overnights are not viable not because of breastfeeding not because my pad is some sort of opium den!
I am of course champing at the bit for when he can come and stay, but I am also aware of the wider issues. Me and Mum have to get on so we can co-parent effectively. We are going to have different parenting styles, and that has to be accepted rather than a source of niggles.
That said if I was in his situation and needed to live with people who did drugs I would understand 100% if my ex found the idea unpalatable. You both ideally need to be in a position where you can express if something is a real red flag issue without it devolving into an argument.
If you'll suffer one bit of advice, if he's at all like me he may be frightened that you'll never want to to relinquish baby for overnights (which to be frank is actually understandable for a Mum for the first year or two). So try and approach the subject that your pleased him and baby are bonding (which I'm assuming he is if he has access twice a week!), and try and stress your objection is not to him, but to the drug use at the house. If you need to, maybe point out how would he feel if the situation was reversed, and you were living with people who used drugs.
I hope it works out for all concerned.