It's a bit of a shame that what could have been a sensible discussion about pool safety has turned into this. Am tentatively creeping in while you are all asleep to put my 2 cents worth in.
In case the OP is still around, I would strongly suggest fencing your yard. I know it's nice to live in a neighborhood where the children can mingle freely like this, but as this issue of the pool has highlighted, it's not always for the best. And it's not just pools but, as has been discussed, ponds, chemicals etc in other peoples yards that pose a risk. Even access to the road by a neighbors yard. I know you said you don't let your 3 year old over to the neighbors in question, but we are all human and accidents can, and unfortunately do, happen.
I am in Australia, and have just had a pool installed. The regulations here are very strict regarding fencing - as they should be. The onus here is on the pool owner to properly secure their pool, and there are penalties for non-compliance. Any pool capable of being filled to a depth of 30cm of water or more is required to follow the pool fencing laws, which include the fence being at least 1.2m high (1.8m if it's also a boundary fence), and having a self closing child resistant gate.
As extra precautions, I also have a pool safety net (which is different to a normal pool cover), and am getting an infrared pool perimeter alarm, which alarms if anyone is detected in the pool area, not just if someone opens the gate. For days there are people using the pool, I also have a bracelet type monitor for my youngest DD (15months) that sets off an ear piercing alarm in the house if it gets wet. She also has swimming lessons. I am competent in CPR and have a refresher course every year. None of this, however, negates the strictest of parental supervision at all times. You just can not be too careful when it comes to backyard pools, or water safety in general. While having a pool can have huge benefits for the family, I never underestimate the dangers involved.
It is pretty much drilled into us not to swim alone (though of course each adult, if not cognitively impaired, is capable of weighing up the risks/benefits of this for themselves) and to only 'swim between the flags' at the beach.
While we may not be the experts, water safety is taken very seriously here for a reason; we sadly have many drownings each year. I think, as with anything in life, if you can learn from others mistakes - why wouldn't you?
BTW, I don't really buy into the whole 'I did so and so when I was young and I survived'. We used to pile in the back of our dads station wagon unrestrained when I was a child - wouldn't dream of letting my own children do the same. 'The absence of accident does not imply the presence of safety' (OH&S saying, finally comes in useful for something)
Finally, I am intrigued by a comment I read earlier on - and I'm paraphrasing here - 'Unlike Australia, children in the UK are supervised closely' (?). I am loathe to go back through the whole thread to see exactly what was posted, or who the poster was, but as I said I am honestly intrigued by this comment, and the basis of it (personal opinion/observation, documented fact?). Is that the general consensus? Could someone enlighten me at all?
TIA & sorry for the essay