Be kind to yourself as Prf says. No one should feel foolish for trusting someone who was supposed to love them. It reflects well on you. Of course you're trying to justify it, and we all, strangely, aim for justifications that are to do with our own actions, because at least we have the capacity to change them.
I think it's unlikely he'll say that to you tomorrow, Starting. Not because he shouldn't, he bloody well should! But because he's spent a long time thinking about this, hence doing it so quickly and having somewhere to go afterwards. He's had a long time and someone to talk it through with. You're having to deal with it in one night out of nowhere on your own. Hardly fair.
The drifting apart will have been more on his side than yours probably and it will almost certainly have been linked to him looking elsewhere, even if at that point it was only emotional. If he was open to that there was absolutely nothing you could have done about it. Nothing. He lied to you when you asked what the problem was because he wanted to sort it out himself, with her, until he was ready to leave. Not very kind to you and not giving you any opportunity to engage with the reality.
Your instincts over the last couple of months have been proved spot on and you have learned they are strong and trustworthy. Have faith in them, don't let him push you off the scent ever again.
He won't tell you anything about her because it disempowers you. Affairs often happen for years before a partner leaves. Was there someone from the past, back when you first noticed a disconnect between you, someone he mentioned a lot then stopped mentioning? Probably someone he works with? It doesn't really matter I suppose but you must be able to see how he's controlling the situation. He expects you to have a conversation about the relationship without telling you anything about who she is and how long it's been going on? How arrogant and cruel. I would be tempted to avoid talking to him at all with such limits on the truth. Has he so quickly decided protecting her is more important than respecting you?
if I were you I would consider not meeting him until you feel stronger and ready. He seems to hold all the cards and fuck that for a game of soldiers. It is you who should be in charge. He's behaved very badly. If you do want to meet him and I say again YOU DON'T HAVE TO, he has basically upped and left you in a state, you owe him nothing. But if you do, I would suggest you insist on a neutral place, I know that's going to be very very hard but having him in the house is going to be hard too, and exactly what he expects and wants.
If you want to get the real truth out of him, we can help with that but you might be a bit vulnerable at the moment. You will need to really stand up to him. You need to take care of yourself and put him off if that's what you need to do.