Deep I know what you're saying. It just depends on your understanding of feminism. If you are on the more radical end, you tend to understand feminism as women's liberation from male oppression. Said oppression is something all women have experienced in one way or another and the oppressing class can't be part of a movement to liberate the oppressed class from that; however they may be able to be allies. So, being born into oppression is a part of what women share and men don't, and it is what men can't understand (don't get and don't get that they don't get) - Karen Ingala Smith puts her point across very well here if you want to read more on this perspective.
In the analogy, white people can be allies to black people, they can be anti-racist, but they cannot understand what it means to be black and they cannot 'rescue' or lead a movement to liberate black people as it then becomes meaningless and opens itself up to all manner of further oppressions, well-meaning or not.
On the more liberal end, feminism is about working towards equality between the sexes, and yes, men can generally be a part of that.
Like I said, it's your own personal judgement based on your own interpretation of feminism. If you've never been wolf-whistled at and never felt oppression based on your sex, then I'm really happy for you!