If you trust each other, why is mixing with the other sex a problem?
It's not - I have plenty of male friends and colleagues that I've been drinking one on one with, including in hotel bars when just two of us are travelling.
There's even plenty of male friends that husband wouldn't batter an eyelid at if I carried on drinking with them alone after a work event.
The difference is, if we were to carry on after an event it's in the context of being drinking buddies already. Whereas this seems to be the first occasion the married colleague in question has asked the OP to join him for one on one drinks, and it's just too pre-meditated to be anything other than a play. Remember this isn't after work drinks, it's after drinks drinks.
(Incidentally, more than one of the married male colleagues that I thought were just mates has hit on me. I told them very clearly where they could shove it, and also told my husband, who knows he can trust me. But both those friendships are down the tubes because the men showed clearly they were not my friends at all, just hoping for a cheap shag.)
So that's a lot of words to say basically - there's going out for totally innocent drinks with male friends, and then there's this.