I've seen it from both sides.
I became involved with someone who was married with kids. I was also in a relationship at the time. We were both bloody unhappy at the time in our respective relationships but feeling stuck/ like there was no escape and it seemed helpful to talk to someone else in the same situation. On a night out I got very drunk and kissed him (unplanned). It went from there.
It gave me the impetus to leave my partner (which was something I'd been trying to find a way to do for years). Shortly after I moved into my own place, he also left his wife, but went back to her a couple of weeks later as he missed his children. We stopped seeing each other at that point, because it was clear to me the future I wanted - and which I thought he wanted - wasn't there.
More recently, my Ex started seeing someone (while we were together). He made it clear to her he wanted to stay with me, and she was just sex. However she refused to accept this, despite admitting to me this was what he'd told her -she thought she would change his mind!. We split up for unrelated reasons; she then contacted me to tell me all the gory details, because she wanted me to know the truth (subtext - she didn't want me to take him back). She was really quite spiteful and nasty to me (she wasn't exactly nice about him either but I bore the brunt) and continued once we got back together; in the end I had to threaten her with the police because she was saying she was going to come to my workplace. Prior to this she'd been making up fake accounts to message me on social media, or different email addresses.
I'm sure some people will say what she did was no worse than me; I'd disagree. I never contacted his wife, would never do so. Whereas I had this fruitloop emailing me detailing what sex positions they did, about threesomes they'd had and all other kinds of shit.
When I was the OW, I didn't care enough about his wife's feeling clearly. But although some would disagree, I know I never set out deliberately or maliciously to hurt her. He told me their relationship was dead and buried; as someone at that time in a similar relationship I had no reason not to believe him. Honestly - if I'd thought we would never have ended up together, if I thought I was just a diversion...I would never have let anything more happen after that kiss.
Do I regret it? Yes and no. I regret any hurt I caused his wife or kids. But if it hadn't happened I might have been stuck in my awful relationship a lot longer, I don't regret that it allowed me to see a way out I wouldn't otherwise have had.