If the school were regularly insisting everyone attend events on a Saturday and giving no notice, I can see why people would complain.
However, given its a one off and people had a years notice and kids get a lot out of it, I'd be prepared to alter my usual arrangements for the benefit of my child and to show support for the school.
This raises the question of how much/far parents are willing to support their child's school? Stuff that gets offered outside of school hours is offered not because the staff love spending extra hours there, but because it is felt is will really benefit the children and their education. I understand that people are busy and have other commitments and these things aren't legal requirements.....but if a year of notice is given for a once a year event, thise who are supportive if the school will mostly find a way to be there. It will be those who basically aren't prepared to make any little changes or sacrifices or put themselves out for the school at all, and who want their child to have the minimum to do with school possible, who will refuse.
Missing one swimming lesson is perfectly doable. There has been a year of notice and the school clearly values the event, has put a lot of effort into it and thinks the children will benefit or really wouldn't have bothered. You can make an excuse of a prior commitment (even though there was a year of notice) and basically not support the school over this (as I'm sure others might do) or you can just accept that it might not be your favourite way to spend the day, but as this is a one-off you will just suck it up and support the school.
Isn't it good too to teach children that sometimes we do have to or choose to attend things that aren't our top choice of use of time....because we are involved with an organisation or people who are running them, and that supporting organisations or people we are involved with is the right thing to do. Learning that they only have to do things that they fancy doing or are convenient isn't a great lesson to teach kids.