No, GroundHog, it's really not.
Your children won't lose either of their parents. They still get you both. They'll see you separately, fair enough, but they'll also see you happy, and they'll see normal, loving relationships, and they'll get the best of both of you.
If you go back to your husband, you'll be showing them a loveless relationship. Your children learn about relationships by observing yours, and it'll be controlling, and fraught, and loveless. They deserve better than that, and so do you.
I know its hard, and you probably feel very stuck, but staying with your husband will make this harder, not easier. You'll lose your own income, and you'll be stuck.
Of course it's daunting, but it'll be okay. You don't lose family events, days out, holidays: you can still go, and you'll meet someone who makes you feel valued and loved and worthwhile, without being controlling or jealous, and you'll wonder why you ever considered staying.
You husband doesn't want a wife, he wants a slave. He's willing to 'forgive' your affair because it means he'll get complete control of you, and he can quickly put you in a position where you cannot leave. And he knows that you'd rather that than leave - he's relying on your need to have a man around.
You deserve better than this, and so do your children. Your instincts must be screaming at you: listen to them.