The Romans thought native Britons were weird and uncivilised for wearing trousers. At some point they went out in favour of hose and tunic (later doublet) for young men and robes/gowns for older ones. Short gowns/smocks for some physical jobs. When hose and doublets gave way to breeches, trousers were a mark of the labouring classes through the 17th and 18th centuries.
So trousers or trouser-like garments have pretty much always been with us as a practical item - but men who could afford it, and didn't have to choose clothes by practicality, had plenty of other more interesting options until the 19th century.
I blame Beau Brummel. The Dandies may now have a reputation as style obsessed clothes horses, but really they are the ones who made men's clothes dull. No makeup, no wigs, no big embroidered skirts on coats - just plain dark coats, plain trousers, plain boots. All immaculately and expensively tailored, sure, but no fun.