Immigration is not a bad thing. I am an immigrant (to another country, not the UK, so I am an emigrant from the UK - immigrant is not the word for an ex pat if you mean a British ex pat working outside the UK, and you are in Britain, obviously...)
An ex pat is a migrant but not necessarily an immigrant - ex pats are not trying to settle in a new country long term, generally.
There are various types of migrant though as everyone obviously knows - not every migrant is an ex pat, not every migrant is an asylum seeker, not ever migrant is an economic migrant, not every migrant is moving abroad for the same reason etc. Like other expressions, ex pat is short hand for a type of migrant. There are overtones of privilege and they can be distasteful if you view it as ex pat being positive and migrant negative - but that is what is wrong, not the term ex pat in itself. Ex pat is just short hand for somebody on a temporary contract with an international company who has lots of attached privileges and perks usually (along the lines of, but not necessarily, international school fees paid) and crucially who is internationally mobile, moving every few years, rather than making a permanent move.
Whilst I am British living in Germany and don't think of myself as an ex pat, I know people who are ex pats - not using the word would just mean I need to write a whole paragraph to explain that they are here on an international contract for a couple of years, put their kids into international schools because there is no point in subjecting them to the baptism of fire that immersion in schooling in a foreign language can be when they will be moving on again soon enough, and they are likely to go where the company sends them next and move every few years for a while before returning to the UK. Most ex pats do also have higher incomes in return for the inconvenience and some skill set their company must find worth retaining and moving about.
In fact I have stopped bothering much with ex pats because they move on and I'm staying. I haven't stopped bothering with immigrants/ migrants, because I am one, and DD is one, and I have more in common with my Polish friend who is married to a German and has bilingual kids in the local school system and no plans to leave but still sometimes misses things from Poland than with British ex pats these days.
I'm not going to stop using the short hand phrase ex pat just because a few people have chips on their shoulder about it, though I acknowledge there is a history of imperialism that you could choose to focus on in order to argue to "ban" the word.
I read an interesting article ages ago by a Mexican-American ex-pat (immigrant to America as a teen, American ex pat in his 40s) which put it all much better, but damned if I can find it...