It sounds like she is dreadfully unhappy and it seems a shame for her problems fitting in socially to be damaging her education the way it currently is. Taking her out of that environment so that she can learn without all the drama seems like it would be worth a shot.
I think though you do/she does need to get to the bottom of her unhappiness and social problems if they persist. Not by forcing her to interact with other teenagers daily if thats too much for her at the moment. But perhaps considering counselling, going to her GP or her confiding in an older person who is sensible but who she trusts - maybe a young aunt/older cousin, etc.
For instance, I wouldn't be too surprised if she was self-harming, or considering doing so if she was talking about it. She may well be depressed rather than just a bit unhappy. There are all kinds of things going on for teens - my younger brother self harmed before coming out about his sexuality for instance. He found counselling helpful with other things too, about his social dynamics with friends and family.
School can be a pressure cooker in terms of social interaction, with small eccentricities or social mistakes amplified. Are there are other things she does, clubs/sports/interests where she could interact with people who have a similar interest, where she is only there for an hour or two at a time, where she could build up her social skills?
Given her age, you and she also need to consider what she wants in terms of qualifications and experience, what she'd like to do when she's an adult so she can work towards it. It might be difficult for her to think about her future when she's in "survival" mode just trying to survive each day of school, but something that she will need to consider.
Is home schooling a practical option for you as a family? I wouldn't consider that its running away from her problems - there are other ways of tackling them - such as attending counselling sessions to work on the root of her low self-esteem, working on her social skills by joining a group/class/club/team, and taking responsibility for her own education, figuring out her goals with you and working towards them with your support. I think you would have to be clear that it wasn't about giving up on either her education or her social skills - instead its an opportunity for her.
One thing worth pointing out here is that you have tried encouraging/forcing her to stick out being at school in the hope that immersion in the school social environment would help/solve the problems. But it hasn't, even with a new more supportive school. At the moment, she's not facing up to her problems or working on solving them, she's just existing, trying to survive, and that "fight, freeze or flight" mode is not good for learning, even without the challenge of dyslexsia.
I wasn't homeschooled. I went to school. I still ended up with huge social anxiety, depression and agoraphobia when I tried to go to uni that cause me to drop out on medical grounds and ended my formal education. I don't know if being homeschooled would have helped - my low self-esteem was partly due to my dad bullying me at home - but school wasn't able to prevent me from being socially inept just by virtue of attending. I kept my head down (generally in a book) and just got through the hard social stuff doing as little as possible, I'd say I was teased and felt like an outsider, but was lucky to have a few friends and wasn't targetted by any bullies at school for more than a few weeks.