Ok, some "tough love" here.
but we fell in love
love, shmove. Love is a temporary condition. Being in love doesn't last. If you google history about love and romance, you'll see it has often been described as disease of the mind. You know, he fell in love with his ex wife, too, you do know that, right? He fell in love enough to MARRY HER. Think he'd ever marry you? Even if he did, you know the saying "if he'll do it 'with you', he'll do it to you". Men like him chase the 'high', and once marriage and domestic drudgery settles in, he goes looking the next 'high'. That was you. He doesn't love you. He was, like all cheating men, looking for excitement and a 'high'.
But the highs never last. The infatuated love 'disease' calms down into a more mature love and acceptance. And many people (especially men) can't handle that so look for the next high. And he will. Again. You were a distraction from his more comfortable and settled marriage. You caused his marriage to break up. A broken home with a daughter he rarely sees. And do you really want to be with a filthy cheater 'oh but it was wuv!!'. Bullshit. It was selfish excitement and a high. Any man that would walk out on his PREGNANT WIFE is nothing but scum from the gutter. He is garbage. Such low standards and taste in men you have there. He is no prize and if I were you, I'd be too ashamed to be seen with him. I'd be too embarrassed by him knowing what scum from the gutter he is.
If you have any human decency, you will leave him and encourage him to re-unite with his proper family. His family will never accept you, and you are in for a miserable life with them in your lives. Is that what you truly want? What misery. For all sides, you've created. It never ends up well, and he will do it to you. I PROMISE you that. And all for what? So you've broken up a marriage, taken a father from a little girl. That is NOT 'love'. You don't love him. That is not love. That is selfishness love high, a temporary mind disease. You don't know what love is. You have a temporary 'high'. Love isn't selfish. Love doesn't tear families apart. Love doesn't have a piece of shit man cheating on his wife when she is PREGNANT WITH HIS CHILD! Love does not take a little girl's daddy away from her. You don't love him. Because what you are describing is not actual love. Love isn't selfish like that. You don't know what love is. If you'd love him, you'd send him back to his ex-wife to re-kindle their relationship.
For that little girl's sake.
Please do it. Because his family will never, ever accept you, and you, and his entire family are in for a life of misery. Do the decent thing. Please. And you can say 'oh but they're divorced now anyway'. But, you can allow him to get his wider family back. And, you can make it easier for that little girl's mother to want her dad to see him more regularly. Please do the right thing. You know yourself this 'relationship' won't last anyway. Make the decision that will see him re-unite him with his whole family. Please do it. For that little girl's sake, and for his and his family's sake. Because your relationship with his family is a write-off anyway, as is your long term future with him.