YANBU. It's normal and reasonable to have the TV on at 10pm, or to be talking with friends at 2am. It's also normal and reasonable to make a bit of proper noise occasionally, such as a birthday party, though polite to warn any neighbours in advance. None of this is anti-social noise-nuisance behaviour.
A PP is correct that you have to nip this in the bud and establish boundaries from the start.
I've had a similar experience to you, with an unreasonable neighbour, but it's a few years down the line. I'm a middle aged woman living alone with my cat, hardly watch the TV, no massive sound system just a tabletop mini speaker. Rarely entertain more than one friend at a time.
I live in an area with lots of small, narrow terraced houses. We all know the party walls are thin. Nobody complains. Nobody, that is, except for this neighbour, a husband with wife and 2 kids. His complaints started not long after I moved in a few years ago.
Complaints about normal noise like occasional TV, occasional music, nothing out of the ordinary in the context of 99% silence from my side. They also make noise - for example the kids screaming in the plastic pool in the garden all summer, the constant rumble of their outdoor tumble dryer near my back door.
It all came to a head a couple of years later, when I was pottering in my garden shed with Radio 3 on one afternoon, and an opera I liked came on, so I turned the volume up. Neighbours kids were as usual screaming in the paddling pool. Within a few minutes he was calling me over the fence telling me to turn the music down. I said no. This blew his little mind.
"Would you like me to report you to the council for anti-social behaviour?" he asked. I said fine, go ahead. And he did. Of course, nothing came of it and I replied to the council's email advising them that it was a vexatious complaint.
Neighbour then emails me with a 2,000 word account of his grievances over the years and explaining why he is always right. I didn't respond.
I came to realise that it was all about control. He thought, - because he no doubt controls his [very meek] wife and two daughters, - that he could also control me, a single woman. He wouldn't have dared behave like this towards a single middle-aged male neighbour.
To cut a long story short, in the end I had to email him telling him not to contact me directly again, and that any necessary communication should be conducted via his solicitors. Not a peep since then.