Obviously neither of you have to agree to it, but your reasons for not wanting them to stay over appears to come from them not helping out more when the children were small.which seems unfair - they are children's grandparents, not parents.
How old are your children? If old enough to be asked, what would they say?
I have lovely memories of staying over with my grandparents. It was often last minute ad-hoc occasions, and I can still remember how lovely it was. My memory still brings up the smell of my grandparent's bedding on the spare bed, and the joy of helping grandad in the garden or walking to the shops with my nana and being able to choose some sweets for the evening. I even got to go on holiday once with one set, as did my siblings individually.
My dd stayed over with grandparents from being about 3. Initially for a reason, such as us going out, later at her request. She's been on holiday with them and she used to have an annual visit to London with my parents to visit my grandparents, when they were alive. Such precious memories for her,
She's always loved staying with her grandparents. She is now almost 20y and at university. She's been for 'sleepovers' at her grandparents whilst there - catching the train and spending 2 or 3 evenings with them, going out for a meal and, now she's older, sharing a bottle of wine, etc with them.
I think often a stronger bond can form through those slightly extended times together, without parents in the way. And obviously a perfect chance for grandparents to 'spoil' the children with too much sweets/sugar than mum and dad allow, and maybe getting to stay up a bit later for a while as part of the treat.