Big source, as used by Robert Graves for I Claudius etc, was Suetonius The Twelve Caesars. He specialised in salacious gossip and it is a really cracking read in Graves' own translation (Penguin Classics). There is loads of stuff in there about Julius Caesar's notorious bisexuality, Tiberius' paedophile tendencies etc, as Katzguk says, it was quite unusual for patrician men to be what we would think of as straight nowaways. Maybe because polite Roman society aspired to Greek standards of culture and learning, and therefore adopted Grecian mores about homosexual relations.
I did find Rome a bit but I suppose it is good to see some good old class civ on the telly...