Look, speaking as a random member of the British public, I appreciate there is a difference between Muslims and islamists.
However, without engaging in extensive conversation with people I can't tell the difference on sight and I am also aware that the majority of terror attacks are carried out by islamists.
Therefore I do draw back a bit and think about my safety when I see visible Muslims.
(I would never speak to a obviously Muslim man anyway as I'm aware that they have rules about not touching women they don't know and Muslim men have been known to refuse to shake hands or even speak with women in business contexts so I assume that they are best left alone)