YANBU - just the phrase 'Nazi memorabilia' makes me feel nauseous and exhausted.
A stall selling Nazi items, as you describe, is like a stall selling instruments of torture, or chains, branks, etc. used on people enslaved and transported from Africa. It's not just tasteless and offensive - it's way beyond both of those.
Nazism was not 'just another' philosophy or political movement. It was a sustained movement to murder and eradicate entire peoples.
It's true that much evil has been done in the name of religions. But that is not the same as genocide being those religions.
And this:
*Find me a religious or historical symbol that hasn't been associated with hatred or oppression or the brutal slaughter of millions.
Buddha, I suppose.*
is really badly informed. Have you seen what is being done to Rohinga people in Burma in the name of Buddhism? But attacks on Rohinga are not Buddhism. Nazism, on the other hand, was genocide.
I probably sound a bit ranty.
My point, though, is YANBU.
I believe that the sale and display of Nazi memorabilia is illegal in some countries - and for good reason.
OP, did you ask the seller why he had that stuff to sell? It would have been interesting to know.