I don't necessarily think this is right or wrong.
Your children got to spend some time with their grandparents, alone. This may have been by prior arrangement, but maybe not.
I don't really see that as a bad thing. It's important that your children maintain a relationship with both yours and your ex's families, and it's not unusual for that to happen without the parents there.
If the children are settled into a routine, and he really does have no-where to take them, then stick to your guns about more access for the time being. You can look at changing it when they are older or as other circumstances change.
But honestly, as hard as it is, you have to let go of what happens when they are with your ex, over and above ensuring they are safe and well cared for.
Your kids want to see him, he wants to see them. His living arrangements sounds a little tricky but it sounds like his parents are helping to provide a stable base for them.
Obviously given the limited amount of time he has you would think he would want to spend as much time with them as he could, but should that mean his parents/ your children are denied alone time with their grandchildren/ grandparents? And sometimes things do happen which mean we can't be with our kids all the time.
If it helps, I have been in the situation. My ex has my son EOW. He had him on Valentines night. He decided to take his GF out, so dropped my son at his parents overnight so they could enjoy an evening out. He will have been well looked after and loves his grandparents. In my ex's shoes I might have made a similar decision.