Our family had always popped round unannounced to each others - was just the way it was all lived on same street
My sister came down with depression and didn't tell anyone, cut herself off from all the family. The odd time she would answer the door, or phone, she would be incredibly rude. It is such a drain / mood hoover, that's her choice and eventually people just stop knocking/ ringing
My point is OP that your family probably realise that something is wrong, and from what I have read, you are keeping it quiet and IMO it is not the right way to go about things - as if they don't know - then they aren't mind readers, you must tell them about the anxiety - they should understand if they care.
Just cutting yourself off without explanation is hurtful to them....and at some point in the future, when you have severed ties long enough, you may need them and no one be there
Just be careful how you treat people in the name of mental health issues - screaming at them is not the answer at all
You have anxiety, yes, but it does not give you the right to scream your face off because a member of family has made the horrific mistake of knocking your door