I think this makes my colleagues believe they're the more powerful sex due to this
Your colleagues sound like a bunch of twats....
To add to the speculation pool, maybe it's because females are generally the sought after sex, whilst males are the ones picked by the females. It's not particularly relevant in this day and age (as with pretty much everything to do with comparing modern society to the rest of the animal kingdom), but across most sexually reproducing animals it's the males looking for females - either testing each other to see who gets the "prize", or being tested by the female herself (birds, scorpions, etc).
I mean, watch any animal documentary! Fish building weird arse nests to attract a female, birds of paradise doing all sorts of weird shit, lions fighting each other, monkeys fighting each other, elephants fighting each other...it's pretty much always the females minding their own business until the males start getting a hard on.
So a female animal is practically guaranteed a mate, whilst the males have to compete for the right to reproduce. And to ensure it's their offspring, some males will take measures to ensure she doesn't mate with another male (polar bears for instance).
In summary, the female doesn't need to have fertility that lasts a lifetime. Males needs to stay fertile to give them as many years as possible to "win" a female. Females don't need to be able to fight to win a mate, but do need to be in top physical condition to give birth.