Would you consider telling your dh @ginandcv?
I had an EA which developed from what I thought was a friendship. I had never hid the EAP's existence from dh and even told dh that I think was attracted to him, but the EAP was absolutely not my type. Certainly doesn't even hold a candle to dh.
Like you, there was never anything wrong in my marriage, it was just that we had moved, I had just began a new job after being off with a baby and had very few friends. I thought I had got on rather well with someone at work who seem to be on my wavelength - all fake.
It was very difficult going NC with the EAP as we worked together and it became extremely toxic every time I tried to break it off. The only thing that worked was him leaving (or getting sacked) and communication was limited.
In the end, I simply ignored his attempts to rile me again by email, stalking and harassment on my way to work. The EAP thrived on reactions and responses from me which explained a lot of things.
Apart from a period of depression and PTSD symptoms from the abusive nature of the association, I am happiest that I have ever been in my marriage with dh. dh never gave up on me and has never used this awful experience against me.