As everyone has said, do it every day - do the same journey every day, even if it is just around your town. I felt confident driving in my town - my husband said to me to keep doing the same route and to make sure to drive in town at every opportunity - to make it the default that I drove.
I sort of dismissed him at the time, thinking, how can I get better driving in my town - I need to go further afield. But you know what - he was right, becuase roads change all the time and by practicing somewhere you know you are taking away the cognitive load of not knowing where you are going and are able to do things like judging whether 2 cars can get past a row of parked cars without either having to stop, or what to do when students walk out in front of you with their headphones in, completely oblivious.
Once you feel a bit more confident then make a plan to go somewhere a bit further afield on you own, where you have to park. Pick a nice day, pick a non rush hour time and google streetview the route, the lanes you will take and the car park. Plan it all. I found it better when I had pressure - i.e. it was for work - I couldn't just turn back and go home, but you might decide to go to a retail park or something to do some shopping. If you decide after a few tries that you can't park then nothing lost - you have driven somewhere new.
I half and half agree with the speed thing - when you get a bit more confident in the car everything will feel slower so you will feel happier at higher speeds. I find that if I am driving somewhere new I still have to turn the radio off and I still find that things seem to happen more quickly than in places and situations I know - but each one you face will give you confidence that you can manage the unexpected.
Where I would say to slow down is doing maneovers - you think that you are taking an age, but really you are not. Take your time and don't be bullied by people who are waiting to get past. Even if it takes a few goes to get far enough into the space to let them past, so be it. If you let them past and you are not in the space properly, just come out a good bit and straighten up.
Honestly, I am still scared driving to a certain extent. I passed my test 15 years ago and have stopped and started and stopped and started, had refresher lessons etc, but this past year I have made a concerted effort to bite the bullet and do it. It has taken a year to get where I am now, and whilst I don't enjoy driving, I would really miss it if I couldn't do it. You will get there for sure - practice little and often for a few month and see how it goes.