You don't love him. You don't miss him. I know you think you do BUT
You love and miss the person you thought he was before you found out about him having a 16 month affair which he started when his baby was 4 months old.
You need time to grieve for that man you thought existed who wasn't a cheat or a selfish git. You can't have him back, he doesn't exist.
The person who wants to have a relationship with you now (or at least a comfy place to sleep and not to be alone anymore) is the type of person who would cheat on his wife and baby daughter for 16 months, move in with another woman and then decide that the grass isn't greener, flirt with his wife behind other woman's back and try to move back in with his wife, at least until something better comes along.
If you had never had a relationship with him in the past, ask yourself: Is that the person who you want to spend the rest of your life with? (or at least, is that the person you want to live with waiting for the next time something 'better' comes along) Is that the example of a relationship that you want your daughter to grow up with? You want her to believe that women should be so grateful to have a man in their lives that they should put up with cheating, lying gits?
Like the posters above, I don't expect you to sympathise with the other woman. But his treatment of her is similar to his treatment of you and your daughter (and quite possibly his treatment of his mother) because he thinks his needs and whims are more important than that of the women around him.
He's not the man you thought he was. Its tempting to take him back because he looks and sounds like the non-cheating spouse you thought you had. If you do decide to start a relationship with this imposter, then you need to do it with the full understanding that it will never be the same, because he is not that man you admired and trusted.
If you didn't have a history with him, if he wasn't the father of your daughter, is this new, much less admirable human being* be someone you would want to start a relationship with?
*16 MONTHS, Starting from when his gorgeous DD was 4 months old and his wife needed him more than ever! I know we are all flawed, but I don't know how he could ever make it up to you, how you could ever trust him again, how on earth could you be sure that he wouldn't let you down when you need him again.