OP, I have a 4 month old, and I understand the frustration. I didn't want this myself, so from day 1, I've blamed covid as to why I can't have loads of visitors all day long. There's been a bit of huff and puff about it, but generally, it's the best excuse I have had.
About the grandparents, I have made it very clear to both sets that when they come over, they come to help. So I give them baby and go for a nap. Or baby needs feeding can they please go make formula. But they are very happy to help out. Have you asked them for help? Or do they just come to play and go home? I bet if you gave them the baby and went for a nap, the number of visits would drop significantly 😂
But in seriousness OP, your priority is your baby. Be firm, yes this is his nap time. This is his feeding time. If they come late, tough, that was the visiting hours available.
Don't make this stage any more difficult than it already is by just bowing to their demands and leaving you with an overtired, hungry, cranky baby. Dont allow them to be selfish.
Also, complain about how last time they came, the baby was overtired so you didn't get much sleep.. You have learned your lesson, that won't happen again.. Ha. Ha. Ha. And let them get the hint.
Or when you put baby down for a nap, say you're having a rest too. Here's the TV remote.
When you allow the visits to go the way you want them to, they will probably start visiting less, or the visits will be less stressful for you.