Isn't there anyone at school who can help? Class teacher, class TA, whoever else goes with them to the pool (likely to be more than just class teacher due to ratios)?
If you don't want to make it about ex's behaviour, but more about practicality, you could simply say 'As DS goes to his dad's after school on swimming day, is there any chance you could hang on to his kit when they get back from the pool, and I'll come and collect it from the school office? It needs to be washed that night to be re-used, and that's hard for Ex to do. I'll pick it up at 2.30 pm if that's OK'
It may not be school policy to get involved - and tbh I would question that stance, in writing, giving precise dates of events - buit an individual member of staff might well be prepared to do something 'informally'.
Alternatively, if it is feasible for you, deliver and collet your DS's swimming kit to the swimming pool. QWe had parents who sed to do that, when I used to take classes swimming - as the pool was closer to where they worked than the school was, they would bring forgotten kit to the pool, or pick it up before we left if e.g. a child had a sleepover or after school activity or contact with a non-resident parent. As long as we knew the parent involved, we didn't bat an eyelid. They just stood at the pool reception as we went past.