The issue there is then the conflating of all people with a particular characteristic, with the individuals or groups which caused harm, and then the acting upon that conflation to cause negative consequences for others.
Extremism can be a difficult term to use in discussions because it can have quite subjective meanings, and can also be used by some people simply to mean "this group does things I don't agree with" while to other people it may refer to groups with very rigid or politicised approaches who use violence and oppression. If we agree to use it as the latter rather than the former, then we can say that some Muslims are extremists and some extremists are Muslims. We can also say the same for other faiths, other cultures, citizens of other countries, etc. If some of Group A also belong to Group B, it does not mean either that all of Group A belong in Group B, or that all of Group B belong in Group A.
Having personal experience of harm caused by extremists (as I and many others will have) does not justify any of us spreading further harm through actions driven by prejudice.
As an example, a good friend of mine was caused serious physical and psychological harm through the actions of a group which is widely recognised to use violence and tactics associated with terrorism, which were targeted against him and others he was with, because of their perceived ethnicity/faith and political affiliations. He has lasting trauma from this and a justified concern about the group responsible for those harms, but that has not generalised to all people who might share a nationality or ethnicity or faith with that group. His experiences do not mean he causes further emotional, psychological, physical or social harm to others who share a characteristic with the people who harmed him but who do not belong to that group.
In the UK, and elsewhere, there is a widespread assumption of collective responsibility and guilt by Muslims for the actions of other Muslims, and an assumption of uniformity. Islam is a globally distributed religion, with a diverse range of ways in which people practice it. People may be Muslim and follow conservative practices in daily life, while others may be Muslim and follow very liberal practices, or only really get involved for the annual festivals (and the gifts and food), or even be 'secular Muslims' (eg born into a family with a Muslim background but don't practice it, or converted for administrative purposes only but not practicing day to day). Yet all Muslims can experience anti-Muslim hatred, prejudice and discrimination, and the harm it causes. It is as unacceptable in my eyes as antisemitism, anti-Christian bigotry, racism, etc.