"He can talk to me about anything knowing I won't criticise him."
I wonder if you realise how that sounds from the outside.
You won't criticise him. For anything. Seriously, can you not think of anything that he could do that might deserve criticism? I'm not talking about the many things he has already done which absolutely deserve criticism; but is there absolutely no behaviour that you would have to, in all conscience, criticise?
Essentially, you are talking about unconditional love. But the only place that unconditional love belongs is that of a parent to a child. Adult to adult, love should always be conditional. Conditional on mutual respect, mutual support, mutual caring. When one partner loves unconditionally, the other partner often starts to abdicate on the respect, support and caring; because if their partner doesn't respect themselves enough to expect these things, why should they bother? And that partner may have started out as a good person, but put them in the environment of unconditional love and it's human nature to devalue that love.
Anyhoo, that's enough on that problem that you're making for yourself. Back to the thing you're asking about, how to live with the guilt. "I don't want to split up with him but I want to know how I live with what we have done. It is really troubling me and is my reason for posting for advice."
Obviously you're not going to like the answer. Ideally you want to continue as you are right now, but not feel bad about what you and he have done. Either to wipe those memories, or just not feel bad about them.
Well, it can't be done. Guilt is the price you pay for the sin you have committed. (I'm using 'sin' as shorthand here, it's less cumbersome than any other way I could phrase it.) It is your conscience holding you to account, or, if you don't have a conscience, it is your awareness of the contempt of society for your actions. It is the natural consequence to bad behaviour.
In days of old people would try to ease their guilt by atoning for their sin; not only by mending their ways and 'sinning no more', but also making amends to the person they injured, giving away all their worldly goods and entering a monastery/convent as penance, flagellating themselves with whips, etc. Replacing the emotion of guilt with a physical/social/financial discomfort.
Now you are quite clear on the point - you are not going to mend your ways, you are not 'sinning no more' - you are sticking with this relationship. You cannot make amends to his wife and children. It could be considered that returning to this thread is a bit of self-flagellation, because you are getting a verbal kicking here. Does it help? (You're not taking the advice offered, so that's the only thing I can see you as getting from this thread.)
I would wish better for you OP. You have behaved abysmally, you're hell-bent on hitching your dreams to someone who will hurt you - and I really really don't think you've dealt with your self-esteem issues. They are at the heart of your behaviour, and I don't see any light at the end of your tunnel until you address them. 