If you are going to stay with this man I think it is right that you keep him separate from your children as they clearly want nothing to do with him (or at least the eldest two).
But I think you need to understand what this could really mean. He may not be welcome at:
- birthday parties
- family dinners
- weddings
- Christmas
I don't just mean your children's weddings. Maybe your niece gets married - either you will attend alone or your children will not come as they don't want to be at the same event as him.
Maybe you move in together and invitations stop coming because people don't like him or don't want an affair partner at their events and they assume you will bring him as you are living together.
If you have grandchildren one day, your children may not send them to visit as he may be there. They may invite you less as they don't want to hear about him or see him.
Your children may not pop into visit you because they don't want to see him - even if he doesn't live with you, they may not want to risk seeing him if he is visiting.
I don't say this to scare you or as hyperbole. These are real things I have witnessed and they have gone on for decades. I have also seen situations where the children go no contact, particularly after the parent and new partner moved in together. It blindsided a family friend because the affair was six years previous and he thought they had accepted his new partner - no, they hadn't, and when she moved in they cut contact rather than face her.
Your children are under no obligation to ever like or accept this man. They may not ever wish to interact with him. They may choose to meet him a few times, you may think their relationship is progressing, and then they may turn around and say to you "I can't do this and I don't want to see him ever again." The fact is, no matter what you do now or what he does now, it can't change the past and it can't dictate the future.
You are an independent adult and you can make your own decisions. But... so can they. And just as the life choices you've made have made your children unhappy, the life choices your children make may make you unhappy.
I don't say any of this to try and be cruel. But I think you need to have your eyes wide open or it may cause you more pain down the line when unforeseen consequences occur.