I agree with you OP, it’s a safeguarding issue. The whole point of safeguarding is to be preventative, not reactive.
While this man may not have been doing anything dodgy, the fact that you are questioning it suggests that it was questionable, and it sure sounds questionable to me. And let’s be honest here, just because he wasn’t actively engaging the children in conversation, he may well have heard their names, and from what you say he was watching. I mean, an unknown adult in an area that’s closed to members of the public, who is sitting near half naked children, watching half naked children, is going to make themselves conspicuous by their behavior, whether they are up to anything dodgy or not.
Personally I would be making a complaint to the center, because this is a safeguarding failure. Also, if people don’t want other people to think their behavior is dodgy, they shouldn’t act in ways that make them look like it might be.
Honestly, I despair of a lot of the posters on this thread. How can we uphold safeguarding when so many people don’t see a problem with a half naked man being where he’s not supposed to be, watching half naked children who he doesn’t know, and potentially learning identifying details about those children? And don’t give me all this bullshit about can’t men do anything without people thinking they are a pedophile? Plenty of men manage to conduct themselves in a way that never suggests they are a pedophile, even paedophiles, so fuck right off with that.
And, you could bet your boots that if it was another parent engineering a situation where they could conveniently watch their own child from the poolside when nobody else is allowed, posters would be up in arms saying they were entitled and they shouldn’t be allowed.