The guys face was a picture when he realised he had whistled at his bosses daughter which is a clue to how wolf whistling is understood by men - it's not seen as respectful, at the very least...
I'm not sure whether it's helpful to think in this context about whether we, personally feel upset or threatened or not? Rather, it seems more pertinent to consider the social context of the behaviour in order to evaluate whether the OP is reasonable in wishing not to be involved in such situations?
Researchers looking at this behaviour do seem largely to agree that it is usually associated with men performing masculine acts for the benefit of other men. See this for example which includes
^Street harassment occurs because our society has always allowed it and dismissed the behaviour as "men being men", says Hadleigh-West.
"Culturally, men have been indoctrinated into it, and it's been a privilege for them to walk down the street fantasising about women. The culture hasn't checked the behaviour."
Because society has perpetuated this as a cultural norm, men tend to engage in street harassment as a way to prove their masculinity, says Northeastern University associate professor of sociology Kathrin Zippel.
"Often times it's not really about the women, it's just about the men performing masculine acts for each other and establishing a pecking order amongst themselves. What is really going on is the dynamic among men.^
The key here is that it is 'the dynamic among men' which is at play when men wolf-whistle and cat-calling, in which women are props for the performance of masculine identity. Whether I would feel harrassed, or you would, or any individual would or would not feel harrassed is not really salient - because the behaviour itself is not really interested in the feelings of the 'target'. Therefore, I think it's totally reasonable to resent being brought into someone else's bid for masculine identity without consent or consideration; it's reasonable to resent being used as a prop; and that it is irrelevant whether or not other women would 'mind' in the same situation.