Phew, thankfully it's not just me then.
I am happily married. Love my dh, love my kids. But there is a guy I am very friendly with. He's very funny, very understanding, patient, calm etc etc. He too is married with kids and neither of us would do anything.
Sometimes I find myself drifting thinking what if?
Then I shake myself and remember what I have.