OP I almost told you that I would have done the same as you, walked away and then felt annoyed with myself.
But then I remembered that a few weeks ago, at swimming lessons, a man actually pushed me as I went to sit down on an empty chair, and told me he was saving it for his wife.
I think it was the fact that he pushed me that did it, because as he draped his arm protectively over the back of the chair, I sat on it anyway.
I'm not sure which one of us was more surprised at that point. We both sat there very stubbornly for a minute, him with his arm still around me, neither of us looking at each other, until his wife walked in.
At which point he removed his arm and told one of his children to sit on his knee so their mother could sit down. Then we spent the next 30 minutes carefully not looking at each other until it was time to leave.
On my way out, one of the other parents said that he does the same thing every week and it drives everyone mad as it means other people have to stand at the back where they can't see their children in the pool. Which you need to do because another lesson starts immediately afterwards so you can't just ask the instructor how things went, you need to watch the lesson.
Apparently there have been complaints about this family taking up five spaces in the viewing area when their other three children don't even watch the swimmers, they play on phones, and there is a sign asking people not to reserve seats and to have just one parent/viewer on the platform per swimming child. There's another seating area for older siblings, extra adults that can take turns on the viewing platform.
But nobody had ever before just sat down on one of the saved but still empty seats. I got a lot of smiles and nods in the changing rooms that day. I'm still surprised when I think about it and realise it was me, sitting there, with an angry strangers arm around me in a battle for a plastic chair and a view of a three year old's swimming lesson. 