Are you using condoms at all? If so, it could be worth investigating size and fit.
Sometimes it can be too tight and restrict bloodflow to the penis. I have had that happen to me. You can feel very hard and raring to go but soon after insertion you can lose your erection and not feel much sensation.
This then causes the same feelings you have experienced. Trust me it isn't easy.
I think you're being too hard on yourself thinking that it's you and focusing on your insecurities. I bet he was feeling worse.
Things that can have an effect - stress, pressure to 'perform' in the moment, alcohol, not enough sleep.
Also it can hard to have the feeling of 'control' and have to hold off the feeling of ejaculation in order to satisfy you.
Spontaneous or risky sex can help where speed is of the essence. A quick two or three minutes of fucking before being caught or seen could work wonder also.
Next time you see him I would wear something sexy - bonus points if you tease him before with some build up sexting - then wank him off quickly and let him shoot onto your clothes or body. This helps build the sexual connection and lets his mind connect you and your body to his physical sensations in his penis.
Once you've made him come a few times that way, it will become easier for him to come inside you. I can only speak from experience but a bit of dirty talk also really helps me. Nothing sexier than being told to come from my wife. and urging me to get there.
I hope some of this can help you. Bravo for talking about it.