This. I can speak from experience.
Went through a difficult 2 years recently (unprocessed trauma, divorce, bad relationship) and due to bad boundaries/people pleasing being ramped up during that period made friends with 5 people who were just wrong for me and increased my anxiety tenfold with the friendship dynamics.
So what happened? Instead of realising they weren't good for me, I took all that was thrown at me from them until I exploded at each of them in turn.
Result: all 5 are gone from my life, my mental health is much improved (also with help of therapy and lifestyle changes) and I can now look back and realise I shouldn't have been friends with these people.
They will happily tell you I'm the toxic one (5 times, how could I not be right?). And perhaps to an extent I was as I put myself in those situations. But at least I have the ability to self reflect and can move on from it. Also, I have several good friends, some of whom I have known decades. They bring me peace. Those 5 new people did not.
On the other hand I also have had to break with my mother and sister a while back due to childhood abuse.
So yeah, I've broken with a few people in my life. But not on a whim. And I'm surrounded by many wonderful people.
It's not always so simple is it.