We have one of those giant Ikea wardrobes and they look like fitted ones.
It was here when we bought the house and if we move it will be left here. It's too big to dismantle and move.
Maybe he assumed yours would be staying because it's so big and awkward to move.
Is it on the property listing? If it is, and you're not leaving it, you'll need to have it taken off because people will assume it's being left or is fitted.
The man peeped in, and yes he should have asked first, both if the wardrobe was staying and then, if the answer was yes, if he could look inside.
But you say he only peeped in and I don't get how that was an affront to your dignity. He didn't search through your clothes or anything.
And also, if your furniture is laid out in the only possible way, even if the wardrobe isn't staying he might need to open the door to judge the space.
For example he might have been thinking "Our bed will have to go where their bed is, and that means our wardrobe will have to be where theirs is, but if it's the case will the wardrobe door open fully or will it hit the bed?"
That kind of thing.
I know it sounds odd but the way the people we bought our house from had their living room furniture laid out was weird. They had blocked off a door and still had no room, there's only one place in the entire room that the sofa can go, and one of my first questions was "Where do you put your Christmas tree when it's up?"
People think about odd things and he might not really have been thinking of looking in your wardrobe exactly, just trying to imagine his wardrobe in the same space.