My husband and I are retired OP, and even though we haven't been out at work during the day, we like to relax and watch the TV in the evening - probably a soap - or listen to some music with a glass of wine etc, and we often don't eat until 8 or 9 O'clock. But if any of our absolutely beloved children came around in the evening with our adored grandchild, we couldn't just ignore them, and turn the TV or music up louder so we could hear it, and we couldn't ignore them while we relaxed with our wine (maybe some families can do things like that, but not us) as it would seem very rude and selfish of us - whereas in reality it is the visitors who are being rude and selfish...
You said OP that if your little one wanders off, either you or your partner go after them. Would the DC wandering off, walk in front of it's grandparents while they were attempting to watch their TV program, and would you or your partner also walk/semi-run in front of them as well, and while you were all getting in the way of the TV, would you at least all be doing so silently?
To be honest OP, you go around there to chat don't you, and to maybe have a cuppa (with a jaffa cake that you won't let your baby have), and you hope that at least grandma will play with the little one, as you are tired, and bored with all the baby talk and activities?
Unfortunately, as the DGC gets more and more tired and grisly, granddad tries to stop them from whining with a biscuit. He is reduced to that because you are trying to chat to your mum, and your partner is trying to watch some of the snooker, and you and your partner have long since managed to block out the sounds of your tired little one grumbling, but grandma and granddad lost the ability for that technique (well actually I never managed to conquer that with my little ones in the first place!) a long, long time ago.
We are getting on now, we need to be able to relax, and to have the confidence to know that we will be able to relax. I really am sorry to be so harsh OP, and maybe your mum is a Saint and really does love it when you go around there, but if I was your mum I would probably be dreading the evenings and you all turning up again.
Please say something like this to your poor parents,
"partner's name and I have been having a chat, and we think it is better if we stop coming around here in the evenings, and get ourselves and baby into a proper routine during the week. But we obviously still want to see you two, so we were wondering if you would like to take turns hosting Sunday lunch?"
If your parents are still up to cooking a full roast dinner maybe you could suggest that you each (as a couple) cook one once a month, so that you each get a couple of full weekends to do your own thing. You could also sometimes offer for them to go over to your place on a Friday evening, and you treat the 4 of you to a fish & chip supper, and as it would be at your place, the baby could still go to bed at it's normal time, and granddad probably won't go rooting in your cupboards to give his DGC a biscuit.