I'm married with 2 sons. He is also married with 2 children.
He calls me... we quickly move on to non-work topics... We discuss all sorts of things.
I don't think it's an EA... Wouldn't we have had to have spoken about feelings for that to be the case?
Focus on your marriage FGS. Sort your own life out before you consider trashing someone elses.
Do you want to be with your DH, Yes/No?
If no, then leave. Don't leave FOR someone else, you need to leave for yourself if you're genuinely unhappy. The "someone else" is what comes AFTER you have divorced, sorted out the house, the kids, the finances. Once you're single, living as a single person, with your own single life, having split from your DH.... THEN you look for another partner.
If the partner you're looking for is married himself, then you have to go about breaking up his marriage.
If you can do all of the above with a clear conscience then by all means go ahead.