It is never wrong, religiously or culturally to leave a bad marriage or relationship but do so with honesty and integrity.
I think you need to be alone, for a long period of time and then see how you feel and if he has been patient and given you space.
As far as this man goes, he sounds untrustworthy but then you met on dishonest grounds and built your foundation together based on infidelity and that is never a good start.
You really need to leave your marriage, it does not sound good but don't leave for another dysfunctional man, do it for yourself.
As for culture, I'm Sikh, come from a hugely influential family. Married someone suitable who then had 3 affairs and decided he didn't want to remain married.
We have 3 children together. EVERYONE in my community pressured me to stay with him as we don't 'don't divorce. There was no way I was going to stay.
Almost 5 years later I have a new partner who absolutely worships me in every way and I him. He is a wonderful man and we are compatible in every way. He's black.
My point is life is what you make it (to a certain degree) be honest in your decisions, don't compromise yourself and do what it best for you and your children. Your community doesn't have to live your life, you do.
And in case I haven't made it clear, ditch the unstable OM, allow your husband a chance to build a new life, build your own life and THEN think about a relationship, an honest one.