"He blamed it on nerves to begin with but its been too long for that to still be the case"
The problem with ED is that once it starts, even if the initial cause has been rectified (such as nerves), the man can then worry about his performance which consequently affects it.
"Hes always watched alot of porn"
Could be an issue.
"he find its only really works if hes the one to initiate"
Possibly because he's the only one to know when he's least likely to fail. If you initiate it he might feel that he suddenly has to perform and that piles on the pressure.
"This is quite an issue as I have a much higher sex drive and feel like I have to tone it down a bit"
I think mismatched sex drives are a separate issue and probably not much you can do about it.
"I want to be patient and supportive but hes not yet been to the doctors to rule out health problems"
You're wasting your time being supportive of a person who doesn't want to even make the first steps at improving the problem. If he gets treatment, and you feel you like him enough to stick around, that's the time to be supportive.
"He has mentioned in the beginning that he will go and get checked by a doctor though"
So why hasn't he?
"I expressed how I didn't understand how he doesn't run into any problems when hes on his own he just got annoyed at me"
It's a very very sensitive subject for a man and unless it's handled very carefully he's bound to get defensive. As he is able to get arect on his own it does suggest a psychological problem rather than physical. Failure to perform matters a whole lot more when with another person than when you're on your own, so that could be why he's fine alone but it all goes to pot when he's with you.
The mismatched sex drives and the not wanting to address the ED are what would drive me away from someone, not the actual ED IYKWIM.