OP - you really need to get your CV updated and out there. No way you haven't already been noticed in the office. It's amazing how people think they have been discreet but really everyone knows. Thing is, even if you don't do anything now, the effect on your repuation is pretty much the same - men are never judged as harshly - although he might be assumed to be only trying to pull you to get favours at work if you're senior to him. (that makes you look both slutty and an object of pity, only being shagged for the career benefit for someone else).
It's not right that woman usually are judged harder in a work situation for sleeping with colleagues (particularly married ones) but it's a simple fact that in most industries, that's the case.
In the same way if you ever had period pain at work you never mention it, your colleagues who are working mothers will come into work when close to death themselves and will go that extra mile to prove they aren't flaky, if you want to do well long term, you need to be more professional than the boys, not less. You are very, very young, you may not have experienced a lot of the more subtle sexism at work, you might have bought the lie that men and woman are now treated equally in the work place, that's bollocks.
Putting aside the morals of sleeping with a married man, focus on your career path and you need to recognise as a woman, your career is basically made or lost in your 20s, if you want children of your own, how badly your career suffers from that will depend on what level you've got to at that point, what flexibility you've got, how much effort the company puts in to retaining you etc will come down to what you are worth. You can't afford to shove your career onto a slow track for 2-3 years in your mid 20s due to being seen as a bit of a joke in the company. It might be by the time you realise what you've done to your career it's too late.
The phrase "never dip your pen in the company inkwell" comes to mind - that and the one about not pissing on your own doorstep.
Flirtations at work should never lead to one to one lunches and dinners, secret dating and the like. Relationships at work always damage the way you are seen unless they end in a 'happily ever after'. Now you've been linked to one bloke at work (and you will be, people will have noticed!), be linked to another and that's it, game over for your career in that company. If it's the sort of industry where people know each other and gossip, you will be "the girl who worked her way through all the guys at X company" and that reputation will go with you.
Date outsiders from now on.