I can relate to some of the issues you describe, and I am just getting to a point now where I am resolving them with my own DP.
We've been together coming up to a year. Sex started off really promisingly but took a nose dive really early on and I was concerned, to the point of thinking of breaking it off because I couldn't handle a mismatched sex drive after coming out of a sexless marriage the year before. I was so gutted though, he is an amazing man and I had really fallen for him.
Anyway, he had his own issues regarding sex. A previous relationship where he was rejected sexually for two years, had become reliant on himself, plus a fairly heavy dose of low body confidence, not helped by an episode of ED when we had been out drinking.
It got to the point where we would be getting down to it and he would over think and lose his erection. Then he avoided getting sexual altogether. He just got really stressed out about it, to the point where he didn't want it. But I didn't know any of this at the time of course.
We were able to talk about it though. It was really hard to talk about at first but we wanted to work on it because neither of us wanted to lose what we thought we had found in each other. We've taken things really slowly and I'd say now, seven months on, it's getting really good and we're both looking forward to trying new things.
It has been worth working on because he really wanted to change things and overcome things and he has really made an effort. Even to the point of a new health and fitness regime so he feels better about himself.
Another thing I'd say is that he is generally really affectionate, and we spend a lot of time cuddling up. I'm never in any doubt that he fancies me and wants me, even if he isn't necessarily trying to get me into bed all the time. Although increasingly now, he is because he has started to see sex isn't something to worry about.
I know people say new relationships shouldn't be hard work and I agree to an extent. I also think that it is unrealistic to think that people who meet when they are older and have been through break-ups don't come with hang ups and a bit of baggage.
To me, because everything else was wonderful, and we were able to be completely open and honest about it, it was worth working on and I am so glad I did. I really love him.
I think it has to be worth it though.