Why don't you turn this around and visit them with spontaneous visits?
That way you decide when you arrive that is convenient for you and you also get to leave when you want too.
Your Dh needs to tell his parents to back the fuck off, you will see them when you want and not on their schedule. Babies don't stick to schedules.
He could phrase it like we know that you want to see the baby but the way you are going about it means we will just reduce contact as it is stressful.
My PIL have always wanted to be treated as "guests" in our home which after having a baby was hard work. They would never make themselves a cup of tea even when told they can. My Mum on the other hand would come over, make herself, Dh and myself a drink and ask if there was anything she could do which included not judging me for having a dirty cup next to me, she would do laundry, tidy, hoover, anything.
I also still had a key to my parents' house even after they downsized, ie not the house I grew up in, I would let myself in, as my Mum did with my house. She never turned up uninvited though, I knew exactly when she was coming.