Rainbowlou1 - what a turbulent time you are having. Lots of posts with different messages. It's your life and your journey.
I hope you and your husband work it out.
I am somewhat older than and I think you will find that the posters who share my views are also older and have been round the block a few times.
A few observations I would make.
When a marriage goes wrong and something like an affair happens. It is a symptom that things need looking at, discussing, working at. A stitch in time and all that. Just because one person had the affair, does not make it all their fault. Relationships get to a dodgy place with two people it. They have stopped meeting each other's needs.
I can tell you when I was younger, I was in the same position as you. I did what some of the younger posters are urging you to do. Told him to push off. He did. That was twenty five years ago. Not a single day has gone by when I have not regretted that decision. I have had two relationships since, but they did not compare and I ended them. My first husband was the father of my son, he was my family, my first love and I still miss him. Fortunately over the years, although he is happily remarried, we have found a shared friendship and meet with our sons family for Christmasses, birthdays, etc. it took a very long time to get to that stage, but it means a lot to me that we can be friends now. He made a mistake, but he had and has so many good qualities I have never found again. I wish I had been more forgiving and accepted my role in the marriage problems rather than laying the blame entirely at his door.
Another thing I have noticed is that the grass ain't greener. Ok, he made a mistake. You kick him out. Then where are you? I am not sure life is all that wonderful for mothers on their own with children. Financially it is tough, it is lonely and can be bleak. Prince Charmings don't come calling often.
Take your time. Don't act in haste. X