I think men shouldn't stay out of feminist activism. But their activity should be directed toward the other men working in conscious and unconscious ways to defend their privilege. You could for example write to the FA about Richard Scudamore and you'd be hard pressed to find a feminist or anyone who would object to that.
Where men are not welcome is in telling feminists how they should speak to other men, because that is nothing like feminist action and a bit more like, keeping the women in line. Object to sexism and sexist views - yes! TO THE PEOPLE PROMOTING THEM.
In fact, I'm significantly enraged by the fact that more men don't do this. The mass oppression of people in our society on the basis of their birthright is something we should all be working against. But telling those whom you are supposedly trying to empower why their own activities are causing them to remain oppressed is oppression apology. And is disempowering, which thereby reaffirms rather than alleviates their position of powerlessness.