I think its quite possible that there are people perhaps more predisposed towards being comfortable with a more homey life. If you are an introvert, or have Asperger's for example then going out and socialising can take a lot of mental effort, and leave you feeling exhausted.
However, anxiety and panic attacks are treatable. Please get help, it sounds like you are having an awful time and you could recover with some treatment. You are already coping better than you think you are, as you are still going out and facing those fears, give yourself some credit for that.
I was agoraphobic for 8-9 years. I am not any longer. I used to be on a high dosage of fluoxitine for depression. I am not any longer. Recovery is possible. I am well. I enjoy life and can manage social situations even if I'd rather have time to myself or speak to people one to one.
I still probably live life at a slower pace than other people but I don't think that is necessarily a bad thing. I'd probably struggle if plonked in a ordinary full time job where I had to interact constantly, because I do find that draining, but I'm happy working from home in my evenings and weekends and looking after DS.
I think I have ASD and that not knowing or understanding that led to a lot of mental distress. Now, although I still find social situations awkward, I can approach them more as a puzzle to be solved rather than something I should criticise myself about. I've trained myself to live more in the moment which reduces the time I spend on feeling bad about the past or worrying about the future. Mindfulness is sometimes used as part of treatment for depression.
I wonder, if like me, one reason that you find being occupied with your children as not overwhelming is that you have to meet them in the moment, it is where they live, especially while young. Their needs and ideas and actions all interrupt your thoughts which when left to spiral unchecked end up causing you anxiety and panic attacks.
WRT Panic attacks. The book I read which explained how they operate was "Understanding Panic Attacks and Overcoming Fear" by Dr Roger Baker. Whatever you do, you need to understand that panic attacks are not your body out of control. It is reacting in a safe way to a threat, the difficulty, why you feel weird is that you haven't seen a Tiger to run from, fight or freeze. It is reacting to a mental threat created by your thoughts.
Graduated exposure is a good treatment for anxiety. As well as CBT so enthusiatically recommended above. :) You've got this.