Agree with OP.
There are several different strands of immigration, which the public generally tend to lump together as if it was all the same thing:
Skilled migration - i.e. non EU nationals with specialised skills coming here to work and pay taxes - is indisputably a benefit to the economy, yet has been slashed in the last year to satisfy the furious public - this is damaging businesses and the economy. Anyone who says "We should have a system like Australia or Canada" is ignorant of the system we do have, which is more restrictive than either of these countries.
Unskilled migration - where lazy good for nothing foreigners come in and work for nothing or steal benefits - is fictional. There is no such thing. The UK does not "let them all in".
Illegal migration - is a different story and it is arguable that the UK needs to control this better - and the government is making efforts; e.g. tightening student visa programme.
Asylum - asylum seekers aren't allowed to work or claim benefits and I would be hugely ashamed of our "great" country if we closed the door to people who really need help. I hope others, even RobF would feel the same. Our grandparents didn't fight for this country in order for us to turn into miserable bastards who ignore genocide and human rights abuses.
EU migration - like it or not, we are part of the EU, and we are legally obliged to allow all 2004 EU entrants (e.g. Poland) to come into the UK and work here. Same will go for Romanians and Bulgarians by the end of 2013. It's either that or leave the EU. There is also an argument that the UK was too quick to allow 2004 entrants to work (it wasn't a legal obligation - there was a grace period during which restrictions could have applied, up until 1 May 2011).
Wage undercutting - I agree with the OP that the biggest threat is exporting jobs. And the more we cut legal, skilled migration, the more this is going to happen. If employers can't employ good, skilled workers at reasonable rates here, they're obviously going to look at offshoring services, from call centres to typing to construction. It makes sense, not just because it's cheaper, or just because technology makes it quite straightforward in a lot of cases, but because good skilled labour in the UK is simply not that easy to come by. Offshoring services and production has a very detrimental effect on the UK economy - it would clearly be MUCH better to allow employers to hire skilled labour, foreign or not, in order to be able to keep businesses alive in this country. But not a lot of people seem to be able to see that.