It'll be the ED. I suffer from it periodically, and while you can say it doesn't matter, it's still hugely embarrassing for the man.
For me, it's the transition from foreplay to actual penetrative sex that is the point where I'm most likely to lose my erection, and that's without putting a condom on, which isn't exactly the sexiest part of the proceedings.
So it's a huge hurdle to get past. It's the choice of either carrying on with the hand / mouth stuff, and we'll both have an orgasm and be sort of satisfied, or the alternative is try to have sex. Maybe it'll be great, maybe it'll all end with a sad trombone noise, we'll both be left unsatisfied, I'll feel I've let DP down, and worried that she's taken it as a sign I don't fancy her etc etc.
When I am dealing with this issue, there are a couple of things that help.
Men can't really tell whether they're erect without external stimulus when lying down (when stood up gravity comes into play and you can feel the weight of it)
So firstly, lights on helps a lot. Basically my brain is going "Do I still have an erection, shit I think I'm losing it, no maybe I'm not, well, not I've thought about it it's definitely gone.". If theres some illumination then I can just look down to verify and stop the intrusive thoughts.
Second, a tight grip during hand stuff. DP has a tendency to treat my penis like it's a delicate thing, despite me demonstrating to her that you can bash it about. As such she tends to have a loose grip on it. It feels good, but in the dark my brain is going, "why such a loose grip, it's because it shrinking isn't it, bloody useless thing"
I've probably given way too much info that wasn't asked for here, so I'll shut up now!