left things for a couple of months and then started seeing each other regularly once the baby was born
That's so nice of the both of you.
I know the timing of us getting together is bad
It wasn't just 'timing'. You had an affair with a married man with a pregnant wife, for God's sake.
I feel bad about it but we fell in love and they are now divorced
No you don't. Not really.
In fact, the way you've written this is quite cold, matter-of-fact and almost callous. It's 'Oh well, what are you gonna do, eh? How can I make this work for me, now I've made this massive mess'?
You may feel some kind of surface level, shallow remorse, but you don't feel it deep inside, and it's really coming across.
You may feel like you've bagged yourself a prize, but all you've got is a box with poo in it, a relationship that's started with cheating and dishonesty.
He probably would have done this with someone else at one point, no matter how you might have convinced yourself that you were special enough for him to leave his wife for. One day, you will realise that, when he leaves you for someone else as well when you're vulnerable and have lost your shine and novelty for him.