Hopefully he is just bitter and angry and is lashing out.
But, err, no - you should not leave. You certainly don't need to be 'punished' for choosing to end a marriage, or taking any other reasonable, legal course of action. That isn't for him to say, any more than it would be reasonable for you to tell him he can no longer, say, drive his car, because he deserves to be punished for being a poor partner.
By the same token, he will be advised not to leave - and that is fine - you at least seem open to mediation, co-parenting, and the fact that you may be in the house together for some time yet.
The main thing is the welfare of the children - and this I think is where you need, if anything, to be a lot tougher. You are primary carer. You jointly own the house, your asset. No, you should NOT be saying that you don't want any of the equity in the house - you have a responsibility to your children here. Don't let the fact that things are unpleasant at the moment and he is making you feel guilty blind you - you'll certainly feel differently if you let him walk away with most of your joint assets, and in three years find yourself struggling, bearing the brunt of costs for the children, while he pays the bare minimum of maintenance but enjoys the proceeds from your joint asset, maybe even with a new partner to share them. If anything, as primary carer you should and will be awarded the majority of the equity. The best course for long-term co-operation and harmony? Get what you're entitled to NOW - at least then you won't end up feeling bitter and shafted. You will NEED that money.
He has no right to tell you to leave - it's your fucking home as much as his. He has no right to tell you you can't leave with the children if you are primary carer if he is at the same time telling you YOU must leave. He has no right to call you unfair or unfaithful in ending a marriage which has not existed for three years. Tell him all this, warn him that you will not be changing your mind and that you will be co-parents for a long time, and if he wants that to be a positive experience the best thing he can do is engage with mediation and remember at all times that your assets are joint - not his to make declarations about.