Im sorry this happened to you OP, and I'm sorry for the flaming you have gotten on here. You forget how awful being a learner driver is almost as soon as you stop being a learner driver.
I passed my test about 18 months ago when I was 38. I'd had lessons for a long, long time and felt like you - driving was not for me, there were too many things to remember. I found it really frustrating that 17 year olds with half a brain cell could drive a car and yet here was me, an (alleged) fully grown, intelligent, functioning adult with a good job and two children couldn't and would get really defensive about it. I swore I was trying to learn to shut people up, not because I actually wanted to drive.
The instructor was absolutely in the wrong for telling you to drive off and it is his insurance that should be claimed on. It makes me wonder if the wing mirror had an issue with it before you got in the car and he was desperate to get it fixed and used you as the excuse. Driving instructors are regulated and you could consider calling the DVSA (Driving Vehicle Standards Association) and asking their advice or reporting the instructor to them.
A good instructor should also show you how to adjust the height of your seat so that you can see better, learning to judge width of your vehicle is really hard sometimes, even harder when you cant see.
Good luck OP, I am sure you will drive in future and please don't let this put you off . I too debated moving to auto but then someone pointed out we would always need two cars and I'd never be guaranteed a car I could drive through work (we have pool cars) so I gritted my teeth and persevered, despite thinking that it was completely ridiculous to be in charge of several tons of metal moving at speed when at points all four of your limbs are doing different things. I now love driving, which I never ever (EVER!) thought I would. Its still scary, I still have moments where I want to abandon the car and walk away, I still make stupid mistakes. But, I can do things I never thought possible, like drive DCs to the beach or share long drives with my DH. You will drive and I promise you, it will be worth every single second of heartache.