I am a lot less hardline in my views on adultery than most Mumsnetters - I think there are some circumstances - when someone feels trapped in an abusive relationship for example, when it is absolutely and entirely forgiveable. And I don't agree that it's always committed by filthy sex crazed animals with no human feeling whatsoever - sometimes married people fall hopelessly in love with someone new, and yes they should leave their spouse before acting on it, but I don't think it makes them absolute monsters if that happens, awful as it can be for the abandoned one if the cessation of marital feeling has not been mutual.
However, considering OP's "dilemma", I certainly wouldn't think that just wanting some sex with an attractive man, is a good reason to risk hurting someone badly? Given that it's pretty easy for most of us to find a willing sexual partner we are attracted to - why go out of your way to target a married man specifically?
Unless you have that "homewrecker kink" thing mentioned above I suppose - but that could be satisfied through role play to be honest. I've played the part of my husband's mistress/secretary before now for a bit of fun - no need to involve and hurt real actual people, just because the idea might seem a bit forbidden and therefore erotic (to some people - I know others don't find it so at all)....
I think the husband in OP's situation is definitely the one who is mostly to blame - he is the one breaking a promise - but I don't think it's ideal behaviour from the woman either, if she gives in to that temptation. It doesn't even sound like the married man is doing all the usual "my wife doesn't understand me, we never have sex, we lead separate lives..." nonsense.....
So yeah, don't do it OP. Find someone else to shag - there are plenty of willing fellas out there. If you want the fantasy of being someone's mistress, find someone who will meet you in a hotel and pretend he's got a wife at home and that you are doing something really naughty..... Why does it have to be real - what possible difference can it make to you? Unless you genuinely want to think you are (albeit indirectly) hurting another woman. If you do that's not a harmless kink, it's something you quite seriously ought to seek therapy for .....