We had the same problem with DS1. He was huge and his chubby legs just wouldn't fit thru the holes. I think that the best thing is to abandon the seat and use armbands. I think that they feel safer being held and being closer to you. Also, DS1's swim instructor (who is an ex-international swimmer) believes that armbands are the best way to ensure that a child gets the right position in the water, and that they gain confidence. It has certainly worked with DS1, he has always been mega confident in the water and at 3 was happy to go down aquazoom slides on holiday!
Good luck with whatever you decide, but I would certainly think about ditching the swim seat.