Islamophobia implies fear, irrational fear. You may as well say far-rightophobia. It is not irrational to fear aggressive troublemakers from whichever part of UK society they originate, it's how you deal with it that matters.
You say extremists are a small part of the Islamic community, my perception is that far right extremists are a minority, but it doesn't take many extremists from any side to cause a shitload of trouble and they all need to be accountable under the law, without fear or favour.
If you're not listened to, if you're called a racist and a bigot for expressing that fear, it doesn't make it go away but it does become an exploitable weakness for people who can turn it to hate.
This is the internet, faceless mumsnetters can lose their temper with impunity as long as the mods are happy. Irl we do our best to live together well.