Hello, OP, are you me?
It took me a very, very long time to learn how to drive and several attempts to pass my test. I've been driving for a few years now and do the same limited routes (work, groceries, train station, post office, doctor). Everything is like clockwork, including when I change lanes. I too freak out if there is a road diversion. I learned how to drive in the UK and now live abroad which makes things much, much harder. However, if I was you, I would take some lessons and also go around some areas that are close to your usual routes to expand your repertoire somewhat.
Practise the routes with someone with you and make yourself gldrive somewhere every day. When you are driving, say out loud "every time I drive I become a better driver. Every time I drive I become more confident". It works. Refuse to have your confidence knocked if you have a scary moment. Do not let it define you and tell yourself that this will be part of your story when you will tell someone about how you overcame your driving anxiety in the future.
People say that exposure and practise are the only answer, which is mostly true, but you need to change your attitude and make sure that you force yourself not to fall into the trap of thinking 'I can't do this'. I'm getting better and am now driving each day (one of my safe routes) but I don't feel like being sick before I get in the car each morning.
Unfortunately, I am one of the annoying friends who can't go to certain places. People don't understand how this anxiety has absolutely defined my life for so long and if I could get rid of it, I would. It impinges on so many aspects of my life but I am fighting very hard against it. Ive suspected that I have dyspraxia for some time and I think that effects things too. But just keep going OP.
Looking on Google Street View and getting an automatic are great ideas.