What you have to bear in mind about people like this, is that they do not care a bit about whether they upset you or not. So you need to put your 'normal' niceness to one side and develop a thicker skin.
They don't give a shit about hoofing their child over the fence without asking, and leaving you to it - so you need to stop worrying about upsetting them and be very blunt.
Excuse me X, please don't lift Tristan over the fence into our garden without asking first.
I'm sure he does want to play, but you need to ask first as it's not always convenient for us.
If you spot him in the garden because they've lifted him over when your back is turned, then you whip out of the pool, round to their front door and hand him straight back. X we have just noticed Tristan was in the garden. I'm bringing him back as we cannot look after him.
And on the days where you are OK with him being there, if he starts playing rough then you remove him and return him to his parents, and explain that he's been playing up and unfortunately your children are upset and you therefore need to bring the playdate to an end.
Stop making life easy for them - they don't care about the impact to you, so you need to stop worrying about their feelings.