Well, you can. It’s not like there is a ban on it. My mother moved from England to Canada when she was a teenager and still used more of the English wording or “slang” until she died at 59.
But different English-speaking countries develop their own slang and words for things. Even Canada differs from the US. Heck even regions of Canada differ from other regions of Canada, and same in the US. They are both big countries. In Canada we also have influences from many languages on our own language. I grew up in a region heavily influenced by Ukrainian culture for example, and with a Ukrainian stepfather, so lots of my “slang” or words for things has that background.
If you said trousers here where I am people would know it was a word, and they would know what you were referring to but they would think primarily of “dress pants” or suit pants/formal pants, because trousers are pants here (hence panties are mini-pants ha). You would not typically call jeans, leggings, etc pants or trousers here typically...they just get called jeans or leggings etc.
Here for underwear you get: panties, undies, underwear, ginch/gotch/gitch (regional slang mostly in prairies and Ontario), and who knows what else. People like my mother would say knickers and people knows what it means but it’s just not common vernacular. Where we grew up if she said jumper people would know she meant a sweater, but if you went more rural people would assume she meant a deer (because they jump over fences...). But when she asked a classmate for a “rubber” (eraser) as a teen they looked at her like she was nuts for asking in the middle of class for a condom!
Language is a fun and funny thing.