He says he wants to stay
Of course he wants to stay and keep his nice easy life with a wife at home, a warm bed and his kids, who his wife mostly deals with.
(just sex, meant nothing)
That means he threw your marriage away and betrayed you for nothing except an easy shag he didn't have to pay for. How wonderful. He's also saying this other woman meant nothing, either. Like a blow up doll, I suppose.
he hasn’t said he felt about my feelings, he just keeps saying he didn’t think he would get caught so hadn’t considered how I’d react.
If this doesn't make you leave him, I don't think anything will. Fuck me, he doesn't give a single fuck about anyone besides himself, does he? He'd still be cheating with her if he hadn't got caught, by his own admission, and I bet he will cheat again.
I'm so very sorry this happened, op, but my marriage would be over if I were you. Who this man really is has been laid bare for the world to see.
He's shit and you deserve better.