I've been friends with a man since we were 16 now mid 40's. We worked together when young. Visited at Uni. Always hung out when home.
Met other halves (me first) at tail end of Uni.
Hung out as a four. Got married. Had babies and due to our friendship- still hung out but less regularly.
It dwindled. I know his wife didn't like me. I suppose I know he fancied me when young. It wasn't reciprocated. I like him, his family, even his wife.
Years pass. His wife had an affair. He took her back.
My husband had an affair and we divorced.
More years pass. We'd have lunch a couple of times a year. Didn't bring families together anymore as I was single, more recently he said his wife doesn't like me (code for she knows he used to like me).
More life passes.
Lately our paths have naturally crossed again - we work close by each other.
We have both been hurt by our spouses having affairs, yet we are not affair people.
But then there is the huge great big But.
I'm struggling and need a big fat dose of MN glaring. Give me your best shots 
I can talk to him about everything except the elephant in the room. Which is an increasing tension. It isn't even sexual. It's more born of loneliness. Well, for me anyway. I know he is lonely in his marriage.
There are children. It's about 8 years until they all have left home. No way would I put mine through any more trauma.
Right. Got that off my chest. Might actually get some sleep tonight 
I need to stop seeing him don't I.