Yes. I have concerns about 2/4 grandparents, but unfortunately they're one each either side, and I'm not 100% confident that the other grandparent would be suitably cautious about their spouse (MIL and my dad are trustworthy, FIL and my mum are not).
For example, I left my 9 month old crawling son in my parent's house with my mum whilst I nipped to the loo. Came back to find him sitting alone in a room with four open doors off it off to other unsafe areas. She'd fucked off for no important reason, and didn't return for another 10m, no idea I'd come back to him.
And at the ILs, FIL gets stuck into jobs, especially in the garden. Also gets very fixated on what he wants to do, and even if a room of people are all saying, "no" or "stop the baby getting at the knife" etc he'll almost pig-headedly keep doing the thing he's doing before attending to my son.
Trouble is, I don't think my dad or MIL would think that they'd be a risk.