You say that this is the first time in your marriage that you have allowed yourself to acknowledge that someone is interested in you. Why is that, do you think? Something has changed. You can't go back to what you had before because you've now opened Pandora's box, even if you don't take things further with the man at work.
So you're now faced with building a new reality. How you do that is down to you.
Sadly, the most likely outcome for people in your situation is that you'd take the route of an affair. Despite the subterfuge, stress, guilt and spectacular fallout after discovery, people persist in believing that this is easier than calmly telling their partner that it's over before involving a third party. Sometimes, people who are so browbeaten by a dysfunctional marriage remain in some sort of paralysis and can only leave because of an affair - the so-called 'exit affair'. It still has the same fallout though.
If your H has genuinely been a poor H for many years, neglecting you, hurting you and placing all the responsibility on your shoulders, you don't owe him the chance to turn things around. If it's genuinely too little too late, don't be guilted into staying while he makes his last-ditch attempt to talk you round. Why now? Is it perhaps because he senses you are close to leaving him finally? What does that tell you about how much effort he'd make if you weren't contemplating leaving and how does that make you feel? If the marriage is making you thoroughly miserable and only getting worse, leave. There is no shame in that. Life is too short to spend it married to a man who makes you miserable and treats you appallingly.
OTOH, if you're only now choosing to see his shortcomings, like many betrayed spouses he's suffering the indignity of fighting to rekindle feelings you have already transferred elsewhere. Even if he's an arse, don't use it as an excuse to behave as badly as he has. It will only destroy your self respect ultimately. Leave instead.
Often, the fear of leaving is worse than the act, and once you've taken that first step, things take on a momentum of their own. If you have genuine fears about how your H may behave, by all means plan your exit first. In fact, assuming your H has been the primary earner all these years, I'd advise you to do just that - obtain copies of all important legal documents and bank accounts, seek some free legal advice, think about where you're going to live and how you're going to pay for it, see what benefits (if any) you're entitled to, CSA payments, etc.
Having an affair, or even just daydreaming about it, is not the solution though.