I started saying 'no' to everything extraneous to my own benefit about two years ago when I was knocked back from a promotion that I should've walked.
Any "opportunity" that comes along, I ask myself 'how will this benefit me?' or 'will I enjoy this?' and I make my decision based on that. I absolutely only do work now when will directly benefit me/my profile or that I'll find fun.
What I've come to realise is that nothing happens, no-one forces you to do stuff, no-one tells you off. In fact, the opposite - you get pegged as a difficult person, someone who's a bit uncollegiate and you don't get asked to do stuff. It's wonderful.
Don't get me wrong, I'm not uncollegiate - I do my admin role and I do it very well, I take my turn doing viva/upgrades, I have a decent number of PhD students, I do sit on committees etc. But I only do these things when I see a benefit or where I think they'll be fun.
My standard response to an "opportunity" is the classic MN - 'That won't work for me at the moment'
or I'll say 'I don't have capacity for that right now'. No need to explain, no need to apologise. Men don't.
On the odd occasions that people have come back and asked why I don't have capacity, I always make sure I have a robust answer back. Often I will draw on the years I spent saying 'yes' to things. For example, when I was two years into my lectureship, I had about 10 PhD students, some of whom were incredibly difficult cases. If someone asks me to come on-board as a second supervisor (and it's not a project which is beneficial/interesting to me), I will say that I had a period where PhDs took up far more of my time than was reasonable or fair and I'm still playing catch-up.
The other option, of course, is to say 'yes' to things and then just do them really badly such that you never get asked again. That seems to be a tactic widely used by my male colleagues.