I think "the love of your life" is kind of a meaningless phrase really.
What's the definition? A lot of people like to think that there's some kind of destiny or fate involved in finding someone to spend your life with, but personally I think that's a load of rubbish. Most people have millions of potential partners out there, and it's just a case of finding one who's not a dickhead who turns up at about the right time.
DP is the love of my life, because we've spent 20 years together and raised a human to adulthood together. But that love exists because we built it, not because of some connection that popped into existence the moment we first met.
But it wouldn't have taken much for that love to never have existed. Neither of us were particularly looking for a serious relationship when we got together, and had DP not gotten pregnant 2 months in, neither of us probably would have grown up enough to make the relationship work long term.
Equally, maybe I'd have spent my life with my second girlfriend, had she not gotten a job half way across the world.
Anyone can say "Oh, my ex was the love of my life", but you have no way of knowing that. Had you actually spent your lives together, you may have ended up hating him for his drinking problem, or his skidmarks in the toilet, or his crappy jokes.
Love isn't something that just exists, there isn't one "right" person out there. Love is something you build, with someone you liked who was in the right place at the right time. And you just have to hope it doesn't get torn down by circumstances beyond control or your own stupid decisions.