If it's annoying someone, it is a problem. (No, I do not mean that everything that annoys me shouldn't be allowed! I'm talking about deliberate actions directed towards someone else which have no practical purpose other than to be annoying. And double glazing salesmen.)
Why should someone have their evening out (or lunch break, or shopping trip, or sunbathing) spoiled by some stranger constantly badgering them? You assume it's a minor inconvenience but what if it's their first chance for a bit of "me time" in months, they were really looking forward to a bit of quiet from the demands of the family for a change or, the one several people mentioned early on, the chance to catch up with a friend or relative you haven't seen in years - and it's ruined by a persistent arse. They don't have all tonight and the rest of the week to do this, it's a rare treat, which possibly they've had to save up for, and now it's been taken away by a pushy git who thinks they exist for his personal entertainment. No-one's going to claim that's on a par with being violently assaulted because, well, obviously, it isn't. However just because it could be worse doesn't mean it's acceptable. There isn't a jury in the land who would let a burglar off on the grounds that he could have stolen much more if he'd robbed a bank instead. And there's no way I'm going to be grateful to an annoying dick because he refrained from attacking me. The refraining from attacking is surely the bare minimum we should expect. We can then build on that to include courtesy and consideration for others' feelings, or what kind of society would this be? Do you really push through life treading on people's toes because, well, you didn't poke them in the eye did you so what are they complaining about?
Re "pushy chatter-upper vs silent rapist", of course women can be attacked by someone who never said a word to her, just as she can be knifed by some random mad woman. These things do happen and will probably always happen, fortunately rarely. However they are not relevant to this thread. There are wars, famines and other hideous things going on round the world all the time and these are all far, far worse than a lot of things complained about on talk boards. But if the answer to everything was "worse things happen at sea", nothing would ever get debated and society would never improve. So I don't actually get your point about "but nobody got physically hurt". Injustice is being done, misery is being caused, fair game to debate and tackle say I.
Whoo, essay 