Oh bless you, give it another six weeks or so, and you will find that it gets much easier. The colic usually dies down, they smile and are a bit more 'awake'. I still remember how awful the colicky evenings were, three years on. I used to pretty much count down the seconds until dp got home, so that I could hand him the baby.
Every age brings it's own challenges of course, but I found the newborn stage the toughest - I thought I'd made the biggest mistake of my life! You are still getting to know each other, you don't need a routine yet, he's still working out wtf is going on. Read up on the fourth trimester.
The things that helped me were to have a shower before dp left for work in the morning, everything seemed easier with clean hair, and to get outside at least once a day, even if it was just for a few laps of the local park.
If your anxiety is making life difficult though, I would urge you to speak to your GP. A lot of it may be attributable to lack of sleep, I know how difficult I found it to keep things in perspective when I was on my knees with exhaustion. Is your dp/dh helping out as much as possible?