SO - having just read this whole thread...
OP, you don't mention that your son has been invited to swim, therefore this thread is not about whether he should be allowed to go swimming whether supervised or not.
You say this is the kind of pool you climb up into, which means you can't fall in, and would also probably have a cover over it in my experience.
You say that the children have freedom to pass between the gardens so I assumer that all the households are happy with this.
Therefore it boils down to this.
Do you have the kind of nine year old DS who will do as he is told? If so, you are being unreasonable to ban him from the garden. It should be sufficient to say 'Don't go in the pool without permission, and if invited, make sure you let me know first, I want to check it's being supervised. You sister is not allowed to go in the pool at all, so please don't encourage her to follow you to their garden, she is too young to be sensible'. (As a side issue, I would not want my three yo to be able to wander freely like that anyway ).
If you don't have the kind of nine year old who will do as he is told, then you are being unreasonable to let him roam this freely anyway.