Look, your neighbours made the foolish mistake of presuming that you were one of the maybe 2-3% or so of smokers that actually give a fuck about not impacting others with their habit, and would modify your behaviour if you were made aware that it was significantly affecting them. They were wrong, it happens.
Yes, it's your garden.
Yes, you have a legal right to smoke there.
But if you continue to smoke there, knowing that you're significantly impacting your NDN's enjoyment of and comfort in their home and garden - and making the executive decision to risk their health for them besides - then you really do have a lot of hide being bothered that they're impacting your enjoyment of your cigarette by daring to comment on your deliberate, knowing choice to bother and harm them within your earshot.
Sure, they were a little passive-aggressive about it.
Maybe they're conflict avoidant in general, maybe they're a little gun-shy after any number of previous experiences that we've all had engaging with anti-social smokers which have resulted in defensive/aggressive/abusive reactions.
By all means, you and other smokers like you can continue to indulge your toxic, carcinogenic, asthma- and allergy-triggering habit whilst obeying the letter of pro-social laws and not the spirit (though, let's be honest, I've seen literally hundreds of smokers light up under 'no smoking' signs).
You can continue to swan round smugly declaring, "It is MY RIGHT in MY GARDEN and IT's JUST A SMELL, GET OVER IT!!!", and light up to your heart's content.
...But don't bitch when someone then concludes from your behaviour that you are ruse, thoughtless, selfish and unethical.
Fact:
Of the two parties here, you are the only one who has the ability to prevent further interruption to your enjoyment of and comfort in your property. In fact, you have at least four options available to you:
A. You can just put headphones in, thus succesfully drowning out the complaints and distress caused by your continued, deliberate anti-social behaviour
B. You could mildly inconvenience yourself to indulge your toxic habit instead of your non-consenting NDNs by taking a walk to smoke (I know, wild, right?)
C. You could attempt to minimise the impact to your neighbours by doing the smoking hut thing
D. You could quit, patch, vape, chew etc.
E. (Would be smoke indoors but you have children and that would make you somewhat more of selfish scumbag, so... No.)
Your neighbours, on the other hand, have precisely zero options to prevent you from continuing to significantly impact their enjoyment of and comfort in their home and garden.
They have no choice suck it up and suffer.
Like all of us, once a smoker knows that their smoke is seriously bothering people and decides to be a selfish, self-indulgent, anti-social tool, they have no option but to stay indoors, avoid their garden/balcony, keep their windows closed, turn off their reverse-cycle heating/cooling (and find alternatives for the extremes of summer and winter) and waste money and electricity drying their laundry in the dryer.
...As for your neighbours supposedly 'coming outside just to cough at you' - the implication being that they aren't really affected and are doing it just to spite you and ruin your fun - has it occurred to you that they were very likely sitting inside with either the windows open or the air-con on, and your poisonous, asthma- and allergy-triggering smoke has wafted in and is bothering them, and that they've come out to say so?