There are alot of reasons why people do certain things, everyone is different and some people can live with certain things and some can't.
You need to speak to your dh, however, be prepared for some lies. He may tell you that it was his first time, he'll never do it again, there was no clothes available when he got out of the shower plus many more.
I went through this with my first serious realtionship. I caught him in a blouse and skirt I wore for work, he told me at the time that he just wanted to see what he looked like in it, I thought it strange but didn't give it any more head space. Then like you I came home early and he was wearing a summer dress I owned, he told me there were none of his clothes handy when he got out of the shower
there was 2 more incidents and having spoken to friends about it, I was told by the majority that it was normal and many men do it, so I went along with it, accepted that he liked to wear women's clothes. The final straw that made me leave him was when I caught him wanking in silk knickers (not mine), stockings, suspender belt and bra. He actually wanted to have sex with me wearing them.
For me, this was not what I wanted, I packed my bags, some people can live with it but some can't, its up to you what you want to do. I remember when I was packing and took my suitcase from the attic and there was a massive suitcase there I hadn't seen before, I broke the locks on it and it was stuffed to the brim with women's clothes, underwear and really dirty porn videos.