YANBU, OP. It's not like you were having a wild party in the garden at 2 AM.
Our neighbours mvoed in with a 14 month old and 3 year old - have been here nearly 3 years. I brought up 5 kids under 11 when we moved n here and never had any complaints about noise - and my kids tore around the house and garden, at will. Now only 2 teens at home, and all they do is play football over the road or go on their PS4. So our neighbours have it easy, compared to my original (nice) neighbours.
Yet from the week they moved in they complained about noise. Said we were slamming doors and complained to the council within days of moving in. Council came out to have a look, and told them they were talking rubbish as they had no furniture and no carpets so any sound from our side was amplified, and basically told them to STFU.
From then on, they have been ultra vigilant trying to catch us being noisy. One afternoon the woman's head popped above the fence to tell us to turn off our power washer (It was mid afternoon, broad daylight, and a weekend) because it was waking her baby. Yet the neighbour the other side, who we share an area of garden with (so they're essentially 1M further away from her) uses power tools at 8 PM and she daren't complain - ditto the neighbour two doors down who is always fixing up motor bikes, drilling, hammering. Bottom line is the other 3 houses here all work so can only do stuff at weekends. They're unemployed so can do things when they like - but still try to control us. But only us - they never complain at the other neighbours manly because they're both built like brick shit houses and would knock them into next week. ;o)
We stopped bothering about being thoughtful when the new neighbours - oh so bothered about their child being woken up - were prancing drunkenly around the front garden at 2AM screaming drunkenly then calling the coppers on eachother...
It's been a while since she's dared complain about the noise. We knew she was lying about the kid being disturbed too, as we hear the way she speaks to her kids (well screams the c word at them) and it's obvious her kid's feelings are her least concern in the world.
So yes, usually those who complain about noise are the ones making it.
For a long time, I was very inhibited in my own house and crept around as I was in fact afraid she'd hurt her kids if we woke them up but three years on - we don't give a shit. Neither do the other neighbours.