I've had a similar kind of thing....lived in my terraced house for a year, have a near 4 year old DD who makes the usual amout of noise, never had any complaints. Then we got new neighours and they had a baby at Christmas.
I was shocked at the amount of noise that came through from them, baby always seemed unsettled and cried alot, and still does. I can hear it every time he is crying no matter what room I am in in the house. Nextdoors DH comes in late most nights from work, slams front door, slams hallway door and then slams cupboards in the kitchen.
I have never ever said anything to them about the noise although it does disturb us (their DS wakes me up at 4.45 every morning) because I remember when my DD was small and I would have been mortified if anyone had said anything, I didn't want to put any pressure on them and I didn't know if they were having problems with their DS etc.
A few weeks ago, next doors DH came banging on my door, we had other neighbours over for a cuppa and I opened the front door to be told off about the 'banging' my DD was doing and could we please keep it down as they couldnt get their DS off to sleep. This was about 5.30pm.
After my initial embarrassment I asked him to speak outside as told him in no uncertain terms that firstly, it was rude to come to my door ranting, epecially as he could see I had guests. Secondly, my DD was not banging, she was sitting on the sofa as he could plainly see and thirdly, it is not my problem that his baby won't sleep. If my DD had woken him up, ok I would have apologised but she didn't. She is allowed to make a normal amount of noise in her own home at 5.30pm.
I also added that as we live in terraced houses, it is expected that you will hear a certain amount of noise from your neighbours and you must be mad to think you will be able to live with complete silence all round.
I told him that I would be willing to forgive his rudeness on this occasion as if their baby won't sleep, they are obviously stressed and were lashing out. If he came round again complaining, I told him I will be asking him to keep his own child quiet in the mornings as he wakes us up at an unreasonable hour and to stop him crying from 7pm onwards too, as thats when my DD sleeps. He has not spoken to me since.
In your situation, I would contact the local community police and explain. I have many neighbours who have been in the same situation as you and the community police quickly sorted it out. You are allowed to make normal noise in your own home and should not be subjected to this abuse.