Elvis have you actually seen one so that you know it is moths? It could be carpet beetle - as you may have guessed they don't just eat carpets!
Look around on the floor of the wardrobe for tiny, hairy cocoon cast offs - they crush into dust very easily. You might also see little white caterpillar like creatures - but they are VERY small so hard to spot.
They always tend to appear if there is a birds nest nearby - so are very common in upstairs rooms due to nests in attics or roof eaves. The best way to get rid of them is a good spring clean to start with - EVERY piece of clothing, linens, towels etc in the area MUST be shaken out thoroughly. If you find much evidence of beetles, you might want to spray around the empty wardrobes with a pest control spray (the ones from vets are the best) if you don't mind using chemicals.
Next, ideally, you need to find and remove the birds nest - or else same problem next year. And by the way, it is always a good idea to remove old birds nests which are in unsuitable places - I have it on good experience that in the 2nd year you are likely to have a bees nest in the same place, and in the 3rd or 4th year, a wasps nest.
Bear in mind that once in the house they do have a habit of moving around - often carried on clothing or laundry (sorry if I'm making your skin crawl) and of course they LOVE carpeted floors, so it is a good idea to keep a sharp eye out downstairs as well.
By the way - I used to work in pest control!