I suspect it is.
I made a "funny" comment to a female colleague about the attractiveness of father's at parent teacher interviews - just a throw away line for a laugh whilst waiting.
If I was a man saying that about the yummy mummies it would be very wrong. So I guess it's as wrong to do it in reverse?