Oh OP you are NOT a lost cause- ignore your families comments. There is clearly an anxiety issue here- have you asked your GP for CBT for anxiety/health anxiety? In some areas you can self refer for these services.
In some regards when something feels so terrible that it simply must be more than 'just' anxiety, that is a natural train of thought- but that doesn't allow for how debilitating anxiety can feel and how many worrying physical symptoms it causes.
Did you try the medications for a decent period of time ( several months) before abandoning them? As others have mentioned medication is tricky to get the right balance of, and a lot of anti anxiety of mental health based medicines can cause unpleasant side effects ( which is normal sadly) or worsening in the initial stages before levelling out. Is it possible your anxiety over health related issues made taking the tablets something you associate with 'feeling awful' and it has become a self fulfilling prophecy?
I take a variety of medication for a physical illness and i can attest that most medications come with side effects. I always worry before starting one that i will feel worse but you need to rationalise that if you do, nothing is going to 'happen', you will not die, you can stop taking it if the side effects are really unbearable but it is worth keeping an eye at the end goal, which in your case is trying to get your life back to a place where anxiety is not ruling over every day.
Do you have supportive friends? Your family need reminding that anxiety is a very real ailment.
www.nhs.uk/conditions/stress-anxiety-depression/mental-health-helplines/
Above are some numbers and links you could consider having a look at if you felt like it OP, there is help available and you are so obviously not a lost cause. I also agree that there could be an agoraphobic element to this- home will start to feel a 'safe place' if you are unwell or go out and feel worse, that doesn't mean that it is where you need to remain to fend of a panic attack because actually, you CAN take control of panic attacks rather than work your life around them. The brain is a wonderful thing, it just needs time to unform automatic habits and reform positive ones. It's no reflection on your ability to cope as a person, and you sound like you are trying your best but merely struggling.