I kinda sympathise with your neighbours. At our last flat we had a couple with a 6 month old ish baby move in next door. The flats were mirror images of each other, so the master bedroom backed on to each other.
For months we were disturbed by the baby crying every single night for hours. We told ourselves it was a small baby, nothing that could be done. About 6 months in, we actually moved the flat around so that our bedroom was as far away as possible in the second bedroom, which was half the size, and the master bedroom was our spare room. We could still hear the crying.
The crying continued almost every night, at least 3 nights a week, for over 2 years.
We decided to move after they sent security up to ours for the third time. We had friends over 3 times, that's it. Each time about 6 months apart minimum. We were always in off the balcony by 11 and everyone gone he by midnight, which I do not think unreasonable. They called security because we were too loud and woke their child up. After the third time, we decided to move (we were renting, they were owners). As time went on, their nanny would turn the TV on for the child to watch cartoons at 2-3am, very loudly, and sing to the child, which was more disturbing than the child crying.
On one of our last nights, we got woken up again and the child did not stop crying for around 3 hours. DH knocked on their door, the bloke told him to keep it down incase he woke the baby! He brought the guy next door round to ours, into our master bedroom at 3 am and said 'and you tell me to keep it down?!' The guy was aghast - he didn't realise it was so loud in our flat. We told him we were moving because of it, because at the end of the day, he's not unreasonable to complain about a loud party (even once every few months), but we knew we were the unreasonable ones complaining about a baby crying
We kept our cool for over 2 years after they moved in. And moved from our home of 5 years because of it.
So, yes I think your neighbours are being unreasonable by complaining, certainly by ganging up like that, but prolonged exposure to someone else's baby crying (and it would be more reasonable if it was the baby rather than the toddler, because there's no reasoning at all with a 5 week old), is extremely stressful. I know how unreasonable that's sounds, but put yourself in their shoes. It's difficult. And the fact that they've all clubbed together says to me that they realise complaining about a baby crying is unreasonable so they're trying to show how much it's disturbing them all so none of them look like the arsehole who can't cope with the sound of a crying baby the bad guy in it all