YANBU.
I rememeber seeing one hung up in an upstairs bedroom of a neighbours house once and it really upset me for weeks (I honestly wan't being nosey, the light was on and it was just so obviously visable).
With hindsight I think the owner was most likely a collector of millitary things given other objects on display but the problem is that the swastika is such a potent and unpleasant symbol that I still can't understand why someone would want to display it in their home.
I know it is irrational but it genuinely did upset me to think that it was the other side of our bedroom walls.
I'm sure that given the the very thin walls he was under no illusions as to how we felt. Dh ranted at length and if he didn't hear his tirade, he had to hear the one from our visiting German friend. The flag disappeared soon afterwards. I didn't know whether I was relieved or embarrassed that he had likely heard dh's none too delicate remarks!