Hey, 40 year old man here who has been dealing with this issue periodically since my early 20s. For me it was a relationship in its dying days that caused it.
The next two women I went out with ended up being non-starters, because I couldn't get hard for them. I then had a drunken one night stand with someone, and it happened again. However she was completely unbothered by it, and we proceeded to have about as much fun as you can have without a penis being involved. The next morning we had another go, and all was well.
However, once its happened once, it lives rent free in the back of your head for ever more. So there have been a few periods over the years where the "Melty man" has haunted me. (Sorry, been rewatching a lot of Coupling recently!)
Basically, it goes like this. There I am having perfectly good sex, and out of nowhere the thought pops into your head "Am I losing my erection". At that exact point in time, the answer is actually no. But its becomes a self fulfilling prophecy, because now you're thinking about that instead of the sex you're having, and so now its gone down, and you feel like crap. And then the next time you have sex you know its going to happen, and you're stuck in a self fulfilling loop.
There are things that can help though :-
Communication - Now obviously this one is difficult. He's feeling shame, guilt, that somethings wrong with him. He's not going to talk about it but he needs to. Talking about it makes it become less shameful. He needs to know that you understand that it's not a reflection on you, that you know he still finds you attractive. That penetration isn't everything, and that you're happy to do other stuff if he can't get erect. This isn't a conversation for just after a failed session, when he's feeling at his worst. Go for a walk somewhere (out of earshot of others) and have it then.
Do other stuff - Don't let the night end limply. When he can't get it up, say something like "I'd love it if you went down on me", or lie on top of him and have a good make out session.
Lights on, covers off - As I said, this starts with a mental "Am I going soft". However, its hard to dismiss the evidence of your own eyes.
A firm grip - We can't actually tell how hard we are in the absence of external stimulus. A long drawn out handjob or blowjob where you've got a loose grip on him feels really nice, but it also makes him feel like he's going softer.
Take control - For me, the point where I'm most likely to go soft is when I'm about to start PIV. There's something about those few seconds after foreplay ends and penetration starts, so take it out of his hands. Jump on him, pop it in, before he knows whats happened. (Actually, I've just remembered you said you're using condoms, so theres going to be an inevitable break here)
Masturbation - You mentioned that he doesn't watch porn, so possibly he's not masturbating much. Masturbation is good, in a "See, I can get an erection when the pressures off, there's nothing wrong with me physically at least"
Fleshlight - Again, for me, penetration is the point where things are most likely to go wrong. A male sex toy that he has to penetrate can help with that in the same way as my last point.
Alcohol - A couple of beers can take the edge of, many beers on the other hand are unlikely to be helpful.
Anyway, these are just the ways I've learnt to deal with it over the years. They may help, they may not. The only real way to deal with it is to learn that it's not a big deal, but thats a hell on a thing to do, and may take a long time. I'm still getting there.