As a number of posters have said, there is a huge difference between a woman being pals with a male colleague and grabbing a quick bite/coffee at costa together on your lunchbreak, or just having a casual friendly meet up.
I have a daughter who has half a dozen good male friends (her age - 20's) who she has known since they were 5.
They have videogaming nights together, (at HIS flat/his parents house,) they go to concerts together, and they go to the cinema together. Her boyfriend of 5 years (who she lives with) knows them all. He knew 3 before he met DD and the other 3 since he met her. He sometimes goes out with them as well.
Her boyfriend has 3 female pals he knows from school, who he socialises with sometimes (not as often as DD does as she is a massive social butterfly and he is quite shy,) but there is no problem. DD knows these girls too and was a bridesmaid for the SISTER of one last summer!
DH has female pals too who he spends time chatting to after work, pops and has a starbucks with on lunch break, and who he buys Christmas gifts for. He buys for all 7 women, and they buy for him. I know them all, and they all give ME a card for my birthday, and pool together to get me a little gift.
A sleazy male (MARRIED) colleague blatantly coming onto a (newly) single female colleague, by wanting her to go for drinks with just the two of them after a works seminar, is a massive red flag. He has also said he has a crush on the OP. Jesus, this is so obvious that he wants to get the OP in the sack, that it's laughable.
As a pp said, I think the OP knows this and is loving the attention.
Oh by the way, sleazy men at work don't just come onto single women. I have been married since my mid 20's (over a quarter century,) and just because I had a ring on my finger, it didn't stop the revolting sleazebags in the office trying on, spouting sexual innuendo, and blatantly asking if I want an affair.
YEP, I had been married 2 years, when my (married) line manager said he was bored in his marriage, and he thinks I seem bored with my DH, and his wife is going away with their toddler for a week if I wanna pop round wink wink nudge nudge.... I said 'Not a chance!' and walked off in disgust.
Vile bastard.
And no I wasn't bored with DH, and never said or implied I was! He was just chancing his hand. A few weeks later, one of my colleagues confided in me that this same manager had asked the same of her.... about 2 days after he asked me.
Fucking scumbag. His poor wife. 
He did leave about 6 months later, and after about 5 years, I heard his wife had left him. Good for her.
Re the OP, I would ask him to bring his wife along for the drink after the work's meeting! As many posters have said though, he won't. 