He seems to be saying that because you're an only child you don't understand that with siblings it's natural (?) to turn up unannounced and spend time with whatever sibling's house you've just turned up at.
And it's about him because he's lost out somehow because you've not allowed or discouraged this happening.
And yet you're the one who always hosted at Christmas and Easter, and this was never reciprocated.
You were also very close to one of them who lived in the same street but who sadly died. You hosted (if I've got this right) the after funeral tea in your garden.
You currently see the other two every week, they and your husband like to ramble on about stuff so you leave them to it. You also go out to dinner with them a couple of times a month.
The only bit that's missing is what happened at the weekend.
But after the weekend your husband tells you you're selfish and his sisters should be able to come round any time they like with the kids.
My take is that now you've got the pool, the three of them have been talking and they've decided you're selfish because the kids would love to play in the pool any time they want and you should facilitate that.
So your husband's being a dick, and his sisters want a free pool for the kids that they can access any time.
I expect that you're happy to have them round and playing in the pool, but not any time they like. I suspect that if this did happen you'd never be rid of them on sunny days as they'd just make a beeline to your pool, and I doubt they'd care if you were there or not because they just want access to the pool.
You need to nip this in the bud. 