We did, in cherry blossom season and it was gorgeous.
Places we loved: Nara - the temples and the deer you can feed rice cakes to.
Kyoto - Golden Pavilion, Fushimi Inari Tori-gates temple and woodland, geisha district, markets, zen gardens
Mount Koya - holy mountain - we stayed at a Buddhist temple on top
Arashiyama - fed wild baby monkeys on the mountain (humans stand in a cage and monkeys come to you if they want), the golden galleon lake ride and bamboo forest
The Samurai path with its waterwheels and rice paddies and tiny medieval villages
Tokyo - all the modern robots in hotels, stations, out and about, the art galleries, amazing shopping, parks, Ghibli museum
Hakone for hot springs onsen baths and the sculpture park, the cable car to the sulphur springs
The food is amazing - noodle bars in the geisha district, Korean BBQ in Tokyo, formal 7 course dinners in Hakone, fresh sushi in Tokyo fish market and grilled skewers in Omoide Yokocho, black sesame ice cream in Kyoto market. Even the 7-11 did great skewers and pork steamed buns for a few pence.
I wish we had seen: Mount Fuji closer up; Hiroshima (no one else wanted to go); the art/cat/rabbit islands; the hot springs monkeys.
I massively recommend Inside Japan. We designed our own itinerary and they put it together for us - booked all the hotels, major train journeys and events. This meant we didn't need tour guides and could travel at our own pace.
Best holiday ever.