You want to talk it out with someone who's been there, done that?
Ok I will admit this for the first time in 9 years of mumsnetting.
I was in your sorry shoes once. For four fucking crap years. The excuses I made to myself to justify it. The lies he told his wife AND me. If I could go back to 1990 (that's when it started) and speak to the 24 year old me, I'd tell her to stop it, you will end up hurt, cynical about relationships, your self esteem will rot till you're on the floor, you will lose good friends over it, and at the end of the whole sorry mess you will still be on your own.
A few years after it, a discussion at work brought up mistresses and affairs. I admitted my past. I was asked 'didn't you stop to think of all the hurt you were causing?' And I replied that the only person causing hurt was the lying cheating scumbag in the middle.
So, yes, I've been there and done that and I am telling you to change job, change phone number, and find yourself a man who will love you, take you out and be proud to be out and about with you, going on dates and not sneaking around behind everyone's back.
What you are feeling is not love, it's a mix of lust and competition. If it was a real relationship you'd provably be fed up of the arrogant git by now.