You really have to be more specific about "Fantasize"
Are we talking to jerk off?
Are we talking about passing a fit woman and wondering what she looks like naked?
Are we talking about a female friend, and wondering if you should ask her out, what kind of relationship would you have?
Are we talking about celebrities and famous people, and again, what it would be like to shag them, or what it would be like to be a couple with them?
I mean, if I'm on the beach and there's someone with a great body in a bikini then I'll check them out and think "Nice" in my head, but I won't automatically imagine sex together.
When it comes to jerking off then it's 99% fantasies about my current partner.
When seeing people I know and work with, then occasionally I'll wonder what sex with them would be like, but it's not in any way "I want to do this, this would be a dream come true for me if it happened" but more just random curiosity, the same way I wonder what it would be like to own more than one motorbike, or visit Machu Picchu. I certainly don't spend hours daydreaming about it.