I appreciate this is controversial, but I'm not all for children being exposed to toxic adults who make the RP's life hell. HIs mother is a personality-disordered, poisonous sort (no wonder he too is a selfish, self-centred, self-absorbed, toxic individual) and I don't think it's in the kids' best interests to be exposed to her more than is 'necessary'.
They, ex and his mother and the tart, are rubbing all this in your face - PLEASE don't listen to them. Their opinions are not worth even basic consideration - look what they've done, look how they behave: can you respect the opinions of people who behave like that? Entirely without morals, toxic, sick people. Their opinions are worth zero, precisely nothing.
I'm also not all for glossing over a very serious crime 'for the sake of the children'. Kids have an intense understanding of right and wrong and, although it's obviously not good to diss their parent, imo it is essential that kids know when someone has behaved badly. You can present is age-specifically and, yes, they may 'suffer' (be hurt) by it but that's because it's hurtFUL, what he's done is hurtful; and life isn't all about avoiding suffering, it's about what you do with it; and imo kids need to know that to prepare them for the realities of living ie shit happens.
I'm at the other end of all this carry on (it happened years ago with me) and at the time I bent over backwards to accommmodate appalling ex, and all and sundry poisonous sorts, at considerable cost to myself - and I genuinely think what I did was misguided. My kids have come away with the belief that 1. I'm christ on a cross, that's my/a mother's martyr role and 2. that you can treat people exceptionally badly and that's all fine.
I'm NOT talking vengeance but I am talking having some common sense about it: its not ok for daddy, or mummy, to bugger off with a piece of fluff and split the family. It's not ok for various persons to gloss over this and pit you as the fall guy, forcing you to bear the brunt of it all 'for the sake of the children'.
Do please remove yourself from the orbit of these poisonous people (re how come you know what they think? Because you're talking to them and associating with them). They will shove all this shit down your throat ad infinitim: they are hell bent on justifying what they've done and you are a convenient conduit for that. You don't have to suck it up 'for the sake of the children', you can remove yourself from their toxic prattling. Do get formal access arrangements in place asap the less the better and do please stop talking to them, associating with them, being around them. And don't facilitate extras, either; don't go out of your way to be accommodating 'for the sake of the children'. Your kids need a stable life with at least one parent who is steady - don't let that be him (because he isn't steady or stable).