Yes men are generally stronger but some men aren't strong(er) than some women. Consequently, some women are stronger than some men.
Saying "you're the man of the office..." without considering individual strength (i.e: Has he shown that he's a strong person or is he assumed to be strong just because he's a man?) will put this one man in a difficult position where he has to live up to being this expectation of strength - and I can imagine it would be seen as weak/shameful (in this case) if he isn't.
That and other comments like it (directed at either a man or a woman) are much more damaging than the ignorant "requires a man's/woman's touch", after you've done something particularly "manly" or "womanly".
Although, the 'woman's touch' statement will also have the same pressure if it's said as a form of expectation as well (Eg: "We need a woman's touch around here. Can you take charge of the decorations?" Said to a woman, automatically assuming every woman is a born interior decorator or that it's her job because she's a woman.) Can also imagine the shame or disappointment if she fails at her "womanly" job.
The real damage is in the expectation from others based on one's sex, not in patting oneself on the back for doing something supposedly "manly" or "womanly". In my opinion, it's fine to correct both but only one deserves to be made a big deal out of.