I guess for people whose whole lives are defined by Their Relationship and who tend not to have meaningful interactions with other men, a deep attraction to someone else could be more significant than it needs to be.
But this notion that a bit of escapist lust and daydreaming must automatically mean your marriage is unhappy even if like the OP, you'd always thought it was fine, seems hopelessly romantic and old-fashioned to me. I also think it's positively dangerous. And is probably why you have so many complete fools ruining their marriages as soon as they feel moist between the legs about some other bloke 
I do think some people think lust=bad and so they try to repackage it as being something deeper and more meaningful whereas people who are more pragmatic, less romantic and more comfortable with lust itself have a better perspective about these things.
And people have some very odd ideas about marriages governing every single emotion we feel about every aspect of life, when we don't apply that logic to anything else.
I love my job for example, but it doesn't stop me fantasising about others from time to time, especially if I see an advert or meet someone who's in a role I could quite fancy. I love my house, but it doesn't stop me looking at houses and occasionally getting a bit obsessed about others on the market.
And all of this is especially resonant when just like there are tough times in marriages, there are grotty points in careers, or motherhood is hard, or your house has got a lot of repairs. Few of us would be so daft to think that a bit of wistful dreaming must mean we hate our jobs, our houses or our children and I don't see any great difference between this and our marriages.