John Stoltenberg talks alot in his books and essays about the difficulty of maintaining friendships with men who are sexist when you choose to be 'loyal to humanity rather than loyal to manhood.'
If you really feel you want to challenge these men rather than just extracting yourself from them, I think it's probably more beneficial to find out how other pro feminist men deal with stuff like this.
The context for a woman hearing misogynist comments from a man will be very different from that when a man hears the same thing. Also, doing some research and reading of your own, at least in the first instance, before asking women what you should do shows you aren't expecting women to come up with solutions for you.
Sanjeev, most feminists do take great pains to be clear that they don't mean 'all men' when they say something like 'men rape.' However, genuine pro feminist men really should take it as read that this is what they mean, spoken or not.
In a similiar way,as a visibly white person, I don't feel entitled to 'correct' a person of colour if they say something like, 'I'm depressed by the racist comments of white people.' I have to trust that the speaker doesn't mean all, but take on board that the discomfort I may be feeling comes from knowing that I DO enjoy racial privilege that they can't. So long as that happens, I remain part of the problem as well as the solution. Rather than feeling personally attacked by the comment, I use the unease I feel as a spur to remind myself not to collude with racists, and be more vigilant in identifying times when I might be doing that indirectly.
It's also important to accept that even if you personally haven't experienced alot of overtly misogynist or white supremacist attitudes or behaviour, it doesn't mean one has the right to challenge or even disbelieve the experience of others who have had more such experience. So for example, one may know no men personally that they think would feel entitled to force another person to have sex. That soesn't mean there aren't alot of men out there who do, and the men who just laugh or go along with this are effectively colluding with actions that harm women.