I've been with mine 6 months. He doesn't often climax, but it means he can keep going for ages, which from my point of view is a great benefit, though I realise others might not want it so much. He warned me he found it impossible to cum inside, though as he has two children, he obviously managed it when he was married, though they are now late teens, so a while back. He was delighted the first time it did happen during penetration together, but it's still a rarity.
TBH, it doesn't bother me, but we are early 50s, and having spent the previous year dating quite a few men, it's nice to have a man whose erection is reliable without needing Viagra. And there are other ways of making him cum. Orgasms aren't actually essential. You can have an awful lot of fun without them, and in a new relationship, you should be trying lots of things to see what works for each other, and having lots of fun in the process.
I think if you make it a thing, it will become increasingly difficult, because it becomes a mental barrier. Does he cum through oral or manual stimulation? Fitness does help with erections (you're basically dealing with blood flow and muscle.) Improving his fitness will be good for him in many ways, whether he's with you or not, whether he's sexually active or not, and it's easier to improve and make a habit at 36 than 56...
But you are half my age, and if at some point down the line you were to want children with him, it could be more of an issue. I wouldn't give up on him yet - sex tends to improve as you get to know each other's bodies better and relax more, but if having children will be important, I'd keep it at the back of my mind before i got too invested in the relationship.