If you swim no wider than other breaststrokers in the fast lane, & just about barely as fast, sounds like you belong there as much as anybody, FranK.
I am the fastest in the medium lane or the slowest in the fast lane (typically). I try to go in whichever one is less crowded. Seems fair to me.
Speaking for self, I don't mind being "held up" as long as people create opportunities for overtaking. Most people pause at the wall & look for anyone who needs to overtake (& gesture to allow them to go ahead). Our fast lane is almost 4m wide, set in a circuit this can mean 2 nested loops in the nominal fast lane. Slower swimmers in outside loop & faster swimmers in inside loop. Most slower swimmers get this system & keep to outside loop also occupy a small pool space so others can get by ok.
The random not-so-aware swimmer swims slowly & not near the edge of the whole lane so basically across both outer & inner loop with a really wide scratching kicking breaststroke I want to avoid being in contact with so they are hard to overtake, except when approaching the wall .. only then they may cut the corners and don't look for faster swimmers, so still difficult to overtake unless lane is almost empty FFS.
(Thank you for getting me off my chest)
We have a regular swimmer who swims in the correct circuit direction but otherwise does WTF he likes. Non-stop Slowly almost down middle of fast lane, fast turnaround (never looks for anyone else), his breaststroke is about 2m wide & I can attest that he doesn't trim his fingernails. He's a character & a grumpy old codger, has a beautiful form breaststroke just no power in it. I am perversely fond of him, he's kind of like an irritating pet you look after out of kindness. My benevolence doesn't extend to anyone else, though.