It's not necessarily the reason for the split which hurts the children, but the way in which the split occurs.
So for instance the OP has had an affair but what is relevant is that she potentially wants to continue the relationship with the OM from the outset. When the children get older they will realise that their mum left their dad for someone else, but right now all they might realise is that their parents were together one minute and next thing they knew their mum had left, moved in with someone else and expected them to play happy families with a new daddy. Leaving because you've fallen out of love can be just as damaging to the children if you don't handle it correctly, and yet Mumsnet would support anyone's right to do that.
Affairs are wrong and obviously I am not saying that the OP having had two affairs is ok, it's not. But the affair has happened, it's what happens now that is the important thing with regards to their children.
We don't know that the OP has contributed to her husband's mental illness, that is all speculation on her part. But his insistence that she leave and indeed people's insistence that because she had the affair she should get out is all to do about self projection and takes no account of what is in the children's interests.
If the OP leaves and goes straight to the OM then there is every chance that the children are going to resent that and feel that they've been abandoned. But if the father kicks her out into a one bedroom flat and limits access to the children for whom she's been a primary carer all their lives then the chances are that it's the father they will resent for throwing their mum out, regardless of his reasons for doing so.
When me and my ex split the affair was long over. We both had to stay in the family home to allow time to sort out the finances. I didn't see or speak to the OM again after that. I didn't seek to be with anyone else either. Yet my ex was on dating sites more or less from day one. Had a brief relationship with a married woman. I came home from a weekend away to find condom boxes in the bathroom bin so no prizes for guessing what he'd been doing while we weren't there.
After I moved out he entered into a long term relationship with the first woman he met online. Moved her and her children in after a couple of months, went on family holidays and suddenly she and her children were considered his family while his own children were left confused. He had more children with his new girlfriend within a year. His own children have almost nothing to do with him now and have no relationship with their siblings because of it.
But do keep believing that it's because of the affair that everything ended and that only the affair damages everyone. Regardless of what went on before or after it was all because of the affair. Life just isn't that straightforward.