Tbf you can’t really blame the friends.
Friendship is it two-way relationship, based on communication and depth.
You may have previously explained you have anxiety. but then, if you didn’t at the time, go on to say : I’m sorry I’m having a bit of a bad time with the anxiety, because of this and that, then how are they to know?
You can’t maintain a friendship if no one comes out, never goes out for a drink, or to a party. With no explanation.
or doesn’t phone, crying saying please can I come round for a cup of tea?
I mean, if you don’t give to the relationship, the relationship itself naturally wilts and dwindles. that is true of any relationship, not just anxiety.