FWIW, I read your OP as assuming posters would hate you because you've got yourself involved with a married man, while you're married yourself. However, I am a bit intrigued about why you felt it necessary to comment on your appearance, because really it adds nothing to the OP or our understanding of your dilemma. Maybe something to think about when all this is done and dusted eh?
Other posters have explained why this is a terrible idea, on so many levels, but you CAN retrieve this situation and maybe even some good will come of it.
First off, you stop flirting back and do the grown-up thing and ask to see the man in private. Explain that you were initially very taken aback by his flirting and worried that to knock him back too harshly would damage your job prospects (and if nothing else cools his ardour, the prospect of HR or an Employment Tribunal hearing that, will do the trick) BUT add that you accept you were equally responsible for letting the situation get out of hand once it started. Explain that you don't want to get involved and that this must stop.
Then, sit down with your H and have a chat about how you've been feeling, but again, have a think about why compliments about your appearance are so important to you, especially if your H is otherwise loving and kind. If he's got a bit complacent, tell him. Up to you whether you tell him about this incident, but your aim is to improve your relationship and your feelings of being desired and loved.
We've all done stupid things at some time or another, but I'm giving you some credit for at least thinking about this man's wife and family, as well as your own marriage and your own self. Don't ever be the sort of woman who shits on another people, to get what she wants.