Hmmm. I suppose the main thing is dealing with why he's doing this.
It's immediate effect is to make you feel unhappy, and insecure - that is very crap. If there are problems, he should be taking them to you, talking about them with you - not indulging in some silly fantasy of reliving lost time with his ex. And she's quite right - he does have you and your child to think of. What a shame she, too, appears to be indulging in the same silly fantasy.
What a very silly couple of people.
Returning to ex = deeply foolish behaviour. There is always, always a reason they became and ex.
Not sure that his wrapping the story of his past relationship(s) in a cloud of serecy is too great, either. Combined with this strange manner of problem-solving, it's a bit secrecy-control stuff.
I think there are a couple of ways to deal with it.
You can address it directly, and call his bluff. Does he really want to risk all he has on a fantasy? That could have the advantage of putting it all out there, in black and white. you;d have to be quite brave, and be prepared for the fall-out of him feeling, perhaps, a little humiliated. And that fall-out could last a while. And it would help if you felt quite strong - which you may not at the moment. (His behaviour is not of the kind that leads to feeelings of security).
Another way would be to be more covert - test out the ground. Are thre problems? What are they? How is he feeling?
If you do the latter, I would suggest reading some of WhenWillIFeelNormal's posts about the aftermath of affairs. Yes, I know it's not an affair but she has good insights on how to address issues within a relationship without you becoming a doormat, trying to make everything luvverly for your partner.
Whatever you do, bear in mind that this is bad behaviour, not adult, responsible behaviour on his part. Whatever the two of you do, he needs to do some growing up in the way he deals with stress and problems.