Yeah. I refused to drive while PG once, because (was before the no alcohol at all point), I had had a small glass of wine and was very tired, so didn't feel competent enough to make the journey (was to my parents, then to my sisters and then back to my parents and home again, a good distance at night time!
I think I need practice, and maybe that is the case for lots of more nervous people.
And on the panic attack thing, I had one ON my driving test, or at least the start of one. We had pulled over, I asked the examiner if I could get some gum (it is one of my ways of averting the attack), and told him why, He was not bothered at all. And when I was unable to do the manoeuvre because of on coming vehicles, he let me do it on another car. And I passed. Was sooo scared of my test I wanted to be sick. I then drove to Oxford from south Dorset (including being directed round the one way system by someone who was learning English and did not drive). And then to Cornwall.
I will drive any distance or conditions, I am just out of practice, so the next time I do, I'd rather have company!
I don't see being aware of your limitations as a driver as weakness, rather, I think more people need to be aware of their failings!