I agree with the PPs above that, difficult though it is, you would be better trying to tackle your anxiety, so that, as a driver and car owner, you have the choice whether to go on holiday in the car or to take the train or a coach, if they seem more practical. Some of the people on this thread don't have that choice in the first place, but you do, if only you can overcome the anxiety.
I'm no expert on driving anxiety whatsoever, but with things like motorways, is it possible to focus on all the positives rather than the very few negatives - namely the high speed (which is the same speed as on dual carriageways anyway)? No pedestrians, no crossings, no roundabouts, no traffic lights, no oncoming traffic, very difficult to get lost as very well signposted long in advance, mostly a dedicated escape lane in case you break down, nobody hassling you to get out of their way if you stay out of the outside lane. Town/city roads have none of these advantages at all, and are actually much harder to deal with.
That said, although you want to have your route (and enough fuel!) and make as many plans as you can, when I was a relatively new driver, I found it helpful to figure that I wasn't driving a hundred miles; I was just driving this half mile, followed by another, then another, then another.... and to take things and deal with them as they come.
As people have said, there are lots of great holidays that you can go on without driving, but it's a shame to decide to exclude an extremely useful option that's available to you. As ChiefBabySniffer said, if you had no car, your child would simply accept that's why it wasn't an option to you, but at the moment, they will have picked up from you that cars are to be feared and, maybe, that only mad/reckless/stupid people would use them for travelling anywhere non-local. In five years' time, they will be able to learn to drive, which is scary for everybody to do; think about the additional anxieties that they will have picked up from you. We need to have a healthy respect for cars and a knowledge of how essential care and safety are in using them, but we shouldn't learn to view them as objects of fear.