It is worrying. Although to be honest I'm more worried about the overall condition of the UK/world/west in general.
For me, the "true" far right has always existed and will continue to do so. Nothing will change about that, and I do believe that they will remain marginalized.
But the current despair and complete lack of hope and positivity that I see in the UK, is pushing people to extreme solutions, mostly in a rightward direction, but to a less degree in other left-leaning political directions as well, as people desperately claw for any solution. These people wouldn't normally be far right or extreme at all, but end up so.
If you can get the UK back on a direction that gives people an inch of hope that things will improve, the number of people radicalizing will drop back to its normal fringe level.
(As contrast, I live in a different country these days, which has its own problems... but generally people here are optimistic about the future, and have seen and are still seeing their lives improve year by year. Politics barely gets talked about, and extreme views are so rare, because no one cares. As soon as people have faith that things are getting better, the intense and depressing focus on our problems and radical solutions vanishes from most people, who go back to getting on with daily life)