"No" doesn't need an approved reason.
That doesn't mean dating preferences like this don't arise from social and cultural bias.
We live in a culture where homosexuality is still mocked, still seen by many as a sin or shameful. That bisexual men are promiscuous, unfaithful, or unclean, or not manly, or really they're actually gay but lying about it. We've seen some of those prejudicial attitudes stated outright on this thread.
Those messages pervade and form unconscious, subliminal bias.
Even if you, personally, don't think bisexual men are promiscuous etc, you're still being affected by that biased messaging and that's what's putting you off.
It doesn't mean you're a bad person, or a bigot, or hateful.
We can acknowledge that biases in dating are based in harmful stereotypes AS WELL AS understanding that dating is not an equal opportunity.