Why shop at lunch time? If it's an old person it can often take them most of the morning to get ready to go out ie having a wash, getting dressed. How very inconsiderate of them to get old and for these daily things take so long for them to do, surely they must know that you are due to nip into the shops.
As for me and my children shopping in the middle of the day, yes I do it just to annoy busy "working" people who have just nipped to the shops, after all I could be sat at home watching Trisha but it finishes at 10 and Jenny Jones doesn't start until 1 so that's my only free slot.
If you're that bothered about it I'd suggest that you take you're lunch earlier and avoid all those people who you don't want to shop with or shop on the internet then you wouldn't have to see them atall!