Don't want to judge you OP. When I met dh (actually now stbxh) he was in a relationship (living together). He had a poor track record - a five year marriage (no kids) ended by him and several other live-in gfs discarded.
His gf went away for three months travelling, and he and I were working together. We fell in love and had a short guilty affair. He ended things with her when she got back, but it didn't last and he dumped me and they got back together.
When they split for good a few months later we got together again. We got married and had two dds. But I can honestly say, I think we were doomed from the beginning. That's not to say we had no happy times. We have managed 14 years.
But the early days of a relationship should be joyful and fun, discovering each other, not guilty and fraught and insecure. There's some old cliche that relationships are like bank accounts and you build up credit (happy memories, closeness) in the good times that you need to draw on in the rougher ones. If you start like this, you begin with an 'overdraft'. I did, and I always resented it if I'm honest, though I thought I'd got over it. When things got rough later on, the early traumas came back to haunt us.
Also, let's look at this guy of yours on paper...
You are not the first, he's a serial cheat (my dh wasn't fwiw). That does make a difference. He hasn't done this because you are the love of his life, it's a pattern and a habit.
He's too OLD for you!!! My dh is 14 years older than me so I speak from experience. The gap can feel bigger as you get older. When you are his age, you will likely still fancy guys the age he is now, but he will be twenty years older and feel like your dad.
If you had a baby with this guy it will all be special and new for you. Wouldn't you like it to be with a guy who hasn't done it all before? It's magical becoming a parent; you can't share the experience with this man, he has had it already.
I'm not judging you for being the OW; we are all human and good people can do 'bad' things. Just look at this with cold eyes. This guy is a loser. Everyone telling you to run for the hills is giving you great advice.