Firstly, I think all toddlers go through a phase of this, so you're not alone :-)
DS1 did this and like you, we told him it was unaccpetable and to use be quiet instead. He still uses it on and off but we just have to look at him now or ask him what he says and he apologises (he's almost 4).
I think the problem might be that he just doesn't take your instructions seriously, if you regularly have to repeat things 3-4 times before he listens then maybe he's learned he can just keep on ignoring what you say as there are no immediate repurcussions to your first instruction?
All children have different 'drivers' (e.g. motivated by food, watching telly, playing with favourite toy, goign to park) but as soon as DS1 used shut up, he'd be told the consequences straight away and if he did it again, those consequences would be put into play (telly switched off, toy taken away, no treats after tea, no park, refusal to play with him etc). This always worked with DS1 who has learned that if we say we're going to do something if he doesn't stop bad behaviour we follow through on it.
We basically did a 'one strike' and you're out approach, he'd get away with one saying, but we would explain what would happen if he did it again and if he said 'shut up' shortly thereafter the 'deprivation' would happen.
Toddlers do like to press your buttons and I'm pretty sure the reason this phase has existed so long is exactly because he knows how riled you get.
I sympathise tremendously (esp with the public embarassment)