if he fell or needed the loo myself or another would happily help
While this is lovely of any adult to do for the child, you have to be careful not to put yourselves in a position that any action you take towards helping him, can be interpreted as anything other than 'trying to help the child'
I.e him in other people's flat, that flat owner is opening themselves up to abuse accusations, same with taking him to the toilet. The stance will be that at no time was the child permitted to go into flat X/go with Mrs X by the parents, therefore a gesture that is innocent will be seen as something else.
You have a child in your flat without the parent's express permission...
An adult who the child isn't familiar with taking him to the toilet? Again, opening self up for abuse accusations.
With him falling, if he gets seriously hurt, the come back could be on whoever was present as they weren't "watching him properly"
Child wanders off/goes missing .. why weren't you watching him?
All of these things would probably get resolved, eventually...but does any one need that hassle? I mean, I don't think any adult would stop a child wandering off from a play space if the parents weren't around, but if they know you are usually there to help him, when you are not & something happens...
The parents are unreasonable, but they will be even more so when they have ammunition against someone who is merely trying to help the lad!
I think this needs to be reported & people need to make sure that the parents know (via a letter, signed by residents) that no one is willing to accept responsibility for this poor child in their absence.
Because I can see something going very wrong & those who were helping getting themselves in the middle of a whole heap of shit!