That conversation would fuck me right off.
Cheating or not - who wants to be treated like a patronised child?
I'm all for coparenting for the kids' sake. I had a coffee with a hated XH withy child recently. Even with no threats to make me want to keep my bf informed, I texted him "fun fun fun, say here with twat ex, still child happy and it's only a quick drink".
That's because my bf and I trust each other, don't patronise each other and want to chat about crap that we have to do!
He's accused her of being crazy, and he's treating you like you're going to react crazy 
I know I'm going on... hard to explain why sometimes just words on a screen give you a bad feeling.
I think I'm a bit anti him because his reason (at least as you gave it) for the split sounds like shite.
I just don't buy cosy fast food with someone who has tried to sabotage more than one of your relationships.