Public opinion (mostly Protestant) also brought us the Gordon Riots in 1780. Which then led to further suppression of public speaking etc which the radicals and non-conformists kicked against in the decades to come.
You can't talk about the Gordon Riots without mentioning the Glorious Revolution which cemented the UK as a constitutional rather than absolute monarchy (yay) but also led to laws disenfranchising Catholics (non yay).
But a lot of the rhetoric against Catholics in the 18th century isn't that different to the rhetoric against Muslims now (they hate our freedoms). And non-conformists were trouble too.
And before then in Henry VIII it was Flemish Cobblers who were the target.
We are a funny old country.