My feelings on things like this are from lessons over the years. Allowing the use of drugs even on a purely recreational basis is not ok (if you not comfortable with it) because as with so many things, once you make it ok, you loose total say over it.
Its a bit like a teenager. Be careful what you allow and how early as once you have said yes - you no longer have much control. If you allow chat rooms, dating, clubbing (that kind of thing) then it takes on a life of its own that you dont get to decide on anymore. Once you have allowed something, its not very easy to "disallow" it. You just get to deal with the fallout of another individual human beings behaviour/reaction to it.
IF your dh gets addicted or starts doing it on a regular basis (why wouldnt he if he enjoys it and you are ok with it?) then you no longer decide how much, when, how much money he spends, his mood swings, the impact it has on your marriage/children, etc Oh he may agree with "only on this day..." but then when he is out, at the pub, having a celebration, on a boring day, stressful day, etc. You will only get to deal with his mood swings, his addiction, the loss of money, change in lifestyle, etc (the fallout) and it will be to late to say "Hang on - thats not ok!"
You get to decide where you lines are and what is a deal breaker and what is not based on your set of values, your morals and life experience. This would absolutely be one for me.
I really hope that I have put this across properly as its all clear in my head but Im struggling to get it down on here.