There are a lot of reasons he would be a higher risk than just rich and famous people. Terrorist threat for one given his royal and military status, plus likely white supremacist groups viewing him as a race traitor
Good thing that he has a special arrangement with RAVEC whereby his security needs are kept under continual review and provided he cooperate, all necessary security will be provided.
Also, you don't know what threats people who are 'just rich and famous' face. Women in particular. Claire Foy had a stalker who made her life a misery had to be deported. Taylor Swift faces numerous serious threats. And other royals of a similar rank to Harry go about their day without any need for armed security.
it is clear he does not trust the state to correctly judge the threat. I don't agree with his approach to everything but in his shoes yes I would worry about a lot about the safety of my family, more so than myself.
Events of recent days have confirmed that Harry is not in possession of the greatest intellect and judgement. It's frankly irrelevant what he thinks.
If the assessments of some of the best security experts aren't to his liking, that's absolutely fine. There is no need for him to be in Britain and he and his family are very welcome to stay at home with their rent a guns in California. Or maybe Nigeria, Colombia or Ukraine.