OP I'm going to try and not be too judgemental.
I'm sorry you feel bad about what happened. It's a very intimate act that many people do not wish to participate in. However, there is nothing wrong with it at all when it's part of a loving and consensual sexual relationship.
So why do you feel so bad about it?
I think because it's forced you to think about the emotional and sexual dynamic you are in.
You did something unexpected (even if consensual) and you probably wanted to talk about it.
But he wasn't there. He was with his wife and you didn't like that.
It's one thing knowing (and compartmentalising) that he has a wife and kids - it's another when a specific event brings it home that you are NOT the priority. I think this is the cause of your angst.
So it's a good time to take stock.
Do you really think he loves you?
Will he ever leave his family?
If he does, are you prepared to be a good Step Mother to his kids?
Are you prepared to be in contact with his wife for the rest of your life (through the kids you will be) and be on the end of her distain?
Can you imagine what you will say when his kids say "I hate you - you stole my Daddy"?
Do you think your relationship with him is special enough for your relationship to even survive once the "gloss" has worn off - remembering his wife was presumably special to him once?
Have you really thought about the consequences here?
This is your first little heartbreak over this man. Stay and it won't be your last or the biggest.