op..it might seem like you're 'losing' people left, right and centre at the moment but when that devil on your shoulder starts up with its negative thinking/intrusive thoughts - just remember that you nobody's victim and deserve to be treated with respect and consideration at the very least.
i went through a period of 'losing' lots of 'friends' in my mid 20's....my so called 'best friend' (who i'd known since school and was by then living with her fella) started flirting openly with my bf - in front of her own bf too which didn't seem to bother him. she would do 'small' things like introduce him to our mutual friends/strangers as soon as we'd walk into a room, always HAVE to sit next to him, invite him to her bf's 'lads night out' ...anytime i mentioned i wasn't happy about that i would be told i was being 'insecure' and 'silly' and was 'making a fuss about nothing'. i almost believed it! when i stopped hanging out with her she helped one of her other friends to 'steal' my bf - inviting him to cheat on me wasn't enough, the bitches even made sure i saw their engagement pics! (Suffice to say he cheated on that girl too and they broke up after an expensive engagement party lmao!) i found out that EVERYBODY in the group i was in knew about the cheating but chose not to tell me. i dumped them all.
re family, i have always been an 'outsider' in my family due to having a very different outlook on life and refusing to bow to peer pressure or their social norms. my older sis is a narcissist and very toxic but nobody wants to acknowledge or deal with that. anytime i stood up for myself or had a (normal) whinge or moan about anything i was belittled, ignored, 'punished' and had my personality and life choices twisted around and thrown back at me. like any problems or issues i had were all my own fault....yet if any of them faced the same issues it was sympathy and help all round. again, i almost believed it WAS me who was the problem. i eventually went very low contact with them to save my sanity.
throughout it all i never had even one person who truly understood what i was going through,which made it so easy to doubt my own self alongside the very low self esteem and confidence.
it was excruciatingly painful and lonely to begin with (i actually believed i was the only person in the world who had no friends or family). Until i realised i was always in pain and lonely when in their company anyway....and this low contact 'aloneness' was peaceful. that period of low contact was the making of me. i had nobody to 'please' except myself and the only demands of my energy were those that i chose to take on.
once i was in a strong and confident place i re-initiated contact with my family - but on my terms only. they no longer treat me bad (though i will always remain an outsider in a sense). my older sis i have chosen to go completely no contact with as she won't accept that i deserve to be treated with the same respect and consideration that she expects for herself. i have a large group of lovely 'acquaintances' (not close enough to call them friends), a small group of wonderful friends and one person who is my very close friend and i can share anything with. These people treat me better than my own blood family or ex-friends did and their presence in my life enriches it and makes me happy.
what my waffling means to say 
.....is that don't worry, it's natural and ok to feel scared or fearful of 'losing' people to such a degree, but if you do what feels right for you - then YOU will NEVER lose out or be the 'loser'. your life will only get better. i just wish i had a crystal ball to see my future with back then, i would have saved myself so much emotional and mental angst and torture.