This man knew you were in a relationship when he was involved & having sex with you 20 yrs ago, right?
So, sorry to be blunt, but he (with solid evidence, the solidest) thinks you're OK with cheating.
You may not have been married but you were in a steady, "exclusive" relationship...and cheated. And went ahead and married the guy presumably without ever confessing you cheated.
So he thinks you're up for cheating, and he looked you up & contacted you when he fancied cheating (since you were OK with cheating on your partner …. not a massive leap for a man to assume you're OK with cheating in general/both ways).
It's pretty simple.
You come across as having zero self awareness, zero awareness of how your actions might come across to anyone; but especially to men, who often tend to be less emotional and more "pragmatic" on average.
He thinks you're up for cheating, he thought that was being confirmed when you entertained his contact (esp when it got more inappropriate).
He's now gone "oh ffs, what a waste of time, what a non starter, fk this" when you bailed on what he thought was going to turn into the extra marital sex he wanted.
He's not a nice person.
He wasnt a nice person back then when he (presumably) knowingly cheated with an attached woman..... he's not your mate, he never was.
(As to his reasons for trying to cheat on his beautiful wife and mother of his child ... common enough unfortunately, doubt they're anything original)
Anyway, you're all "what's wrong with men?" .. equally what was wrong with you, that you felt the need to cheat on the man whom you later saw fit to marry, presumably while forgetting to tell him about it (?)
It's pretty ironic).