I think is is, really is, a case of 'the one that got away', I've been there, I know.
I know very well because it's happened to me three times. These ones that get away are pesky, & enduring. But they are boringly repetitive & that's exactly what they'd become.
They rear their heads in times of dissatisfaction & toil, they represent freedom & abandonment. A different life. A more exciting life.
But they're just men, as we are just women, they don't offer the answers, only distraction.
I'm absolutely sure you felt as heady & excited by your DH/P when you met them & if you hadn't settled down to a life of wet towels on the bathroom floor, children in the bed & hairs in the plug-hole, if they'd buggered off & left you for dust a year in, you'd feel a yearning for what could have been.
& what could have been?
Exactly what you have now I imagine, it's not all Liz & Dick out there you know.
Most enduring relationships are comfortable, personable & fun. Occasionally sexy, sometimes wild, but mostly comfortable, understanding, intimate & kind.
You'll never get there if you hanker after an impossible fantasy.