Yes, being inconvenienced in one way does not make it ok for you to inconvenience others.
I could take the same logic and say 'because my house was burgled it's ok for me to burgle a shop, as in my view that's a lesser crime, with less serious consequences'.
A bike is vehicle, they're not allowed on the pavement, it's illegal and antisocial to ride them there and does have a negative impact on pedestrians. That this is a lesser impact than cars have on cyclists is irrelevant, see above. How would you feel if a motorcyclist took the same attitude? They have a hard time on roads too.
I know conditions on the road are awful, scary and dangerous. The only way to ride there is competently and very confidently. Training is available. You can campaign for better conditions, which will take forever to make a difference. In the meantime the choice is, ride your bike legally, or use another form of transport.
I am a cyclist, pedestrian, car driver and public transport user and do sympathise with the road experience. However, as a cyclist and environmentalist I am embarrassed and feel let down by pavement cyclists, who do annoy a lot of people, scare others, come across as selfish and (if using an environmental justification) desparately sanctimonious and undermine the cycling 'cause'.
I've seen so many threads in other places on this topic and it always descends into blatant prejudice, name calling and bickering, so I can't really believe I've just gone and contributed to what will inevitably become such a discussion (go on, amaze me people, don't do it!).
I do feel strongly about this though and see pavement cycling (by anyone over the age of about 14) as further evidence of an increasingly selfish society (I'll do what suits me regardless, my needs and wishes are more important than my responsibilities to others).