Your neighbour smoking dope is the least of anyone's worries really. I mean your neighbours could be doing ANYTHING behind closed doors - smoking dope ain't that bad - really! He might be a bit of a slacker but he's unlikely to be violent or aggressive.
If he is growing, he will not necessarily be selling (as many growers sell it on wholesale basis not wanting to 'deal'). So people coming and going not necessarily an indication of anything.
In my book we need more local people growing in their spare rooms as they do in Holland - it's not dangerous, and it's keeping business local, and we all like local. I'd love to buy weed from a guy like this as would many other people, as opposed to some dodgey drug dealer who is buying imported weed where the money is probably funding criminal activity, taliban (IRA in years gone by), and who probably sells harder drugs too. It's this side of the coin that gives weed a bad name.
Don't worry you won't be getting stoned - and the smell of a room of it growing is kind of funky - strong, sweet, thick, perfumed - I think much more pleasant than cat pee. It could very well travel to neighbours place. You may smell it outside when his doors are open too.
In my book your neighbour is providing a valuable local public service and I wish he lived on my street. Bummer he won't be taxed on the income he will make, that's my one complaint.
But if the smell is bothering you, you could mention it as he can probably increase insulation etc to minimise disturbance to you.