OP, I posted yesterday evening
"So I'm guessing he moved straight in with you after she threw him out?
"
I definitely maybe wasn't specific enough about what that implied question was.
I'd like you to think about timelines and sequences of events. And to be honest with yourself.
Your OP included "It started out as a casual relationship and I had a lot stronger feelings for him than he did me. But we ended up spending a lot more time together and grew a lot closer and our relationship turned serious."
So your feelings were stronger than his. Presumably at this time, his wife didn't know? What came first, his wife knowing or him deciding your relationship was serious after all?
When she threw him out, did he move straight in, or get his own place for a while? Because there's a big difference. I'm guessing he moved straight in (or very very quickly). After all, he strikes me as a man who likes his creature comforts. Someone to do all "most of the household stuff, cooking, cleaning, shopping" so that he doesn't have too. I've often thought that most serial adulterers would prefer to be single and shag around openly, but, well, it's nice to come home to a clean house and food in the fridge without having to put any effort into it, so they marry to get that and shag about not-so-openly.
So, back to the timeline. And be honest with yourself. How close in time was him deciding your relationship was more serious, and her finding out? Or to really spell it out - did him getting thrown out make him pretend to take your relationship more seriously because he needed somewhere to stay that was already furnished?
I know you're still in denial, but he really is taking you for a mug. Yes, you have behaved badly; I suspect foolishly and naively rather than maliciously. Your friend's devastation has taken the wool from your eyes. But in your current situation you are not just a perpetrator, you are becoming his next victim. Step back from it a bit and see it for what it is. He is a user and a taker. He's charming to you right now, on the surface. But sometime soon, if it hasn't already happened, those words 'stale' and 'excitement' are going to resurface.
You need to ask yourself - is that what you actually are here? The supplier of comfortable domesticity, just like his wife was, a base from which to continue his established pattern of cheating?