The Nazis at the time were in the habit of smashing up psychoanalysts' libraries, Lou Salome being a good example. The usual justification for this was that psychoanalysis was a 'Jewish science', and Salome having been friends with Freud wouldn't have helped her case.
In the case of Hirschfeld's institute, the very extensive fetishism and pornography collections would probably have attracted accusations of outraging public decency.
The followers of Wilhelm Reich also got it in the neck, although Reich himself escaped into exile. Reich isn't quite as fashionable as he used to be though, while Hirschfeld has been rescued from obscurity to become the 'trans doctor', and we gloss over the elements of his work that were just as cranky as Reich's stuff.