Ruminating on origins of 24 year relationship

[TheScottishPlay]

Been with DH for 23 years. Married for 19. We were friends due to a shared interest for around a couple of years prior to that during which he had a girlfriend he was besotted with. She messed him around but he kept going back for more until he finally had enough and they split for a final time.
During the time he was with her I developed strong feelings for him, but kept these to myself until the relationship ended. We went out as friends often before and after the end of his relationship and gradually became closer and began to see each other romantically - all was well for a few months then he rebounded badly. His ex became sainted and all aspects of their relationship was recounted to me through rose tinted glasses. I cooled things at this and we reverted to friends. However, during this time he did a variety of cruel things - vocally admired other women in front of me which hurt as of course I still had feelings for him. I asked him to stop doing these things which he ignored. I had a couple of flings during this time which he didn't know about but remained deeply in love with him. He says he didn't sleep with anyone else during this time and he becomes upset and sorry for the way he treated me saying that he had a breakdown following the end of his previous relationship but wanted to be with me and was frightened I would move on without him.
We went on to have a strong relationship and had a decent life together. He says I was the only person he's ever really loved and wanted to live with, marry and have a family with. I told him about the flings I had fairly recently and he was terribly upset. We have had problems caused by his family and his defence if them and an unwillingness to seek treatment for ED for a number of years. In the past 6 months this has been resolved with him acknowledging he handled things with family members wrongly and our sex life has also skyrocketed again.
I just have never shaken off the feeling that I've been 'second best' relationship wise and he loved and wanted her more. He was much more adventurous sexually with her but says he didn't feel comfortable with it and prefers being with me as it feels warm, close and loving.
I've felt this in the pit of my stomach for 24 years and can't live with it for the rest of my life. Despite being deeply in love with him, I'm so angry with myself for staying when things went wrong early on and living with feeling like this.

"second best" to someone who he was in a sexless relationship with?

No, that's not what I've said - his previous relationship was very much based on sex.

@EvenMoreFuriousVexation

"second best" to someone who he was in a sexless relationship with?

Why do people comment without bothering to read the OP

@TheScottishPlay that sounds tough but your worries are ultimately rooted in the past when you presumably had lower self esteem, and you have to consciously accept you made the decision you felt was right at the time. The causes of low self esteem are normally not our fault and are rooted in childhood/parenting experiences. With the benefit of maturity, your DH will now fully see that he wouldn't have been happy with that ex gf anyway and I'm sure is feeling very relieved and happy that he chose you.

Also for all you know, you may not have met anyone who you liked as much as your current DH or if you did, it may not have necessarily led to marriage with that future man. The most important thing now is that he treats you well. Focus on the present moment and forgive yourself for what you regret, and forgive him for past hurts. Share with him how you're feeling and let him acknowledge how badly he hurt you in the past.

Thank you @DateXY. You are right and I needed to read this. Thank you for taking the time to read and reply.

OP - it sounds a lot like a midlife crisis of sorts on your end.
The ‘origins’ of a relationship that lasted for 24years and survived various life stages and issues - shouldn’t be causing you this must ongoing unhappiness.
Young relationships - may seem more intense when looked through the lens of a long marriage. But you can’t compare and judge and get upset over it.
You don’t know how your life would have turned out if you didn’t stay with him back then.

So - I’d say - do a bit of a soul searching - on your own or with a professional. If you are happy with the relationship you do have and see the future and getting old with him - you need to find a way to put away your regrets and disappointments of a quarter century. It’s unhealthy to dwell on the past to that extent.
But - if it’s a way your subconscious trying to justify wanting out of this relationship - it’s your right too. You don’t have to be with him .

@MMmomDD There is definitely an element of this. My DM is at the end stage of her life and I'm spending a lot of time sitting with her reflecting on what I'll be grateful for and regret when it's my turn. I've mainly put these thoughts to the back of my mind but they have been ranking more and more in current circumstances.

I am sorry.
Times like that are difficult and bring up a lot of emotions. I think - some of that is linked to your mind looking for escape from the sadness that is directly in front of you.
You need to get through this first, heal and only then attempt to re-examine your life. If you still feel like it’s what you want to do.
It may or may not be.
Hang on in there.