@burnedoutgrandma We sound a similar age. I was brought up by my mum when my parents divorced and I would say that your expectations over contact with his kids are rooted in the same sort of upbringing as I had.
Sadly, and respectfully, just because that is what happened for you as a single mum, doesn't mean it was right and you are being very naïve about todays fairer, more realistic expectations of shared parental responsibility.
Sadly for you, you made a commitment to let your son live with you and for him to have your grand kids in your home. I can see that with this new realisation, its more than you bargained for but it is wholly reasonable of his ex to ask him to take a fair share of his responsibility whether she works full time, part time or not at all. She probably can't afford to work full time with childcare responsibilities that she carries by herself most of the time. She is on the breadline more likely than not. Being on her own with kids most fo the time is not an easy place to be and you should understand this and support his equal care of the kids. Even if she didn't work at all, he still should have his share of childcare and if that means he needs to pay for clubs when they are with him, then that is what he needs to do.
Having said all of this, I can see how difficult it is for you, as its your home and you are trying to help your son and its harder than you had expected.
But here is the thing, it is his responsibility and so you shouldn't be taking time off to help him unless you want to spend the time with your grandkids. He is their Dad and as hard as it is, he should be capable of looking after his kids for a few weeks by himself, without your help every waking hour.
You have facilitated a place for him to live and a safe environment for his kids to visit which is very generous, but the rest is for him to decide upon and cope with. He clearly wants the kids (credit to him - you clearly brought him up well) hence asking if you would reconsider, that doesn't mean you have to take time off too, has he asked you to, or are you just assuming you are needed?