It is usually easy enough to remove them, I tend to either trim or replace them to keep some shape without looking like an extra from Dallas.
So many blazers have poorly shaped joins at the shoulders, unless you keep your arms down at your sides you get an unsightly crease right at the top of your arm with the shoulder sticking out like you're in a marching band. I hate it.