I agree that often the distinction is about whether the move is likely to be temporary or permanent, and whether a company sent you overseas or you did it yourself.
But many, many (British/US/Aussie etc) people who move overseas having applied for work there themselves, and with the intention to stay long term or to move from one country to another with no plan to return to their home country, class themselves as expats.
I'm British and I teach overseas. Most international school teachers spend a number of years in one country (from 2 to 10+) before moving on to another, then another. Many don't intend to return home and will eventually settle in one of the countries they worked in or choose another place. But they form a considerable chunk of the 'expat community' in many cities all over the world.
Personally, I'm fine with expat or immigrant. I have expat privileges with my job and don't know how long I will stay in any country, but I wasn't sent here by a company and each time I move I do so indendently. But then, I don't see immigrant as pejorative. I do always find it interesting discussing politics with British people overseas and sometimes hearing xenophobic and anti-immigration views from people who are themselves immigrants...
I once had a discussion about these terms with a 6th form class and a Japanese boy said well based on the definition of work etc his family were expats, and it's true that in schools we would consider all the non-local students as expats. But most of them fit the more classic definition of being there for a period of time due to their parents' work.
Having said all this, yes, I do think in many ways the distinction is at least partly based on race (and class and nationality). I do think more 'expats' are becoming conscious of this and use the term immigrant more now.