Sometimes FIL and MIL look after our DCs in our house. We are probably up to half a dozen times, compared to the many times they have done so in their own house.
When they do so, FIL always without fail finds a job that needs doing but isn't remotely urgent and does it. We then get a lecture about not having done it ourselves.
I find this terribly rude. It isn't his house so it isn't his business. Jobs like brushing the patio do not affect the health or well-being of his grandsons, whereas ignoring them instead of playing with them just might.
WIBU to ask him not to do so, but to "relax and enjoy the DCs" or similar? Failing that, WIBU to pull out the old "Did you mean to be so rude?" if he does this once again?