I hate how casual people are about weed smoking, and I say that as someone who has lots of friends who use/used it recreationally.
Someone referred to them as hippies, there's this idea that its people chilling out with a joint of an evening, no big deal, same as the smell of incense. In reality, strains have got SO much stronger. I know two people whose lives have been destroyed by drug-induced schizophrenia, the result of exposure to much much stronger weed in their teens. Others whose lives have been negatively effected. And yes, ones who cheerily enjoy a smoke at the weekend/at a party/when their kids are at sleepovers, but those people are not smoking in the kitchen in the middle of the day while their kids are playing.
So yeah, if we're worried about young children's increased asthma risk from parents smoking tobacco (and I know two friends who reckon their asthma is entirely down to their parents 20-a-day habits as children) then we should be worried about the mental health risk of children constantly exposed to week, which is what the OP is describing. And be as worried about parents driving the morning after as we would be if we knew they downed several bottles of wine between them.
Honestly though OP, if they're nice, I'd maybe mention it to them. They probably won't be aware smoke travels. But the garden will almost certainly be worse. However, there might be gaps. Is there another neighbour who could potentially be to 'blame' if you raise it and then make a complaint? Because if not, they'll probably figure out its you anyway. Another potential avenue if you really do think they're driving high is mentioning that to police, its coming up to Christmas so random road blocks and breathalysers are a thing.
You need to think through the risks. But, v regular smokers are unlikely to quit. They may move it outside some of the time. So the question is would that be enough. If not, you have to weight up whether you want official intervention, and if you do maybe go straight down that path.