So he has them two days out of every fourteen, plus about five weeks in the holidays? Is that about right?
He's doing very little, frankly. A minuscule amount. With that amount of custody, even if he talks to them via video call or messages daily, he's not being an active, hands-on father, developing and maintaining a day-to-day relationship. He's most likely going to become a distant figure they holiday with, and aren't close to in the slightest.
To try to cut down on the already tiny amount of time he spends with them is incredibly irresponsible. I can understand they exhaust you, and it's not what you might want at this time in your life, but surely you don't want to drive a wedge between your son and his children?
In terms of their holidays, I spent almost that much time at my grandparents as a child, with my siblings and cousins! They were practically never without several grandchildren in the holidays. But in your case, if you find it difficult then you need to step away and let your son handle it all. It's not your job, it's his.
If it's not working out, then he needs to sort it. Ideally, he'd be able to get a job in the same area as his ex and the children live, and rent a house there, so he can see them more often. Obviously, that's easier said than done. But it's his responsibility to be part of his children's lives, and you either need to support that, or step away entirely.