What do you mean, how it happens?
Someone dies, leaving an estate and no will. No known relatives - no kids, or kids have died. So the estate is liquidated (house sold, assets disposed of, debts settled) and the money sits there in a pot. It's then listed on the Bona Vacantia list for England and Wales
www.gov.uk/government/organisations/bona-vacantia
Or the QLTR in Scotland.
www.qltr.gov.uk/content/estates-which-administration-has-been-completed-and-which-remain-available-claim
That money sits there, unclaimed, for a set period until it goes to the government. Private companies and individuals can then look at that list and try to track down people who might be entitled to inherit under English or Scots law - cousins, nieces, nephews, half siblings. People who never knew the deceased but may have a legal claim.
All the "heir hunter" companies need is a copy of the deceased's death certificate - £15 from the GRO. That gives them the parents' names. Use sites like Ancestry to track their marriage, then their births, their siblings, their parents. This is all PUBLIC information obtained from birth, marriage and death certificates. Track down the lines, find living relatives using the electoral registers and social media. It's not shady, not dodgy.
Of course, once they find an heir they will say "we think you're entitled, sign with us and we'll put forward your claim for a percentage of the money you get". You are free to accept their help or not. Many people do, because these firms are the experts in providing evidence to the Courts in exactly the format they want, and know how to navigate the system. People who weren't expecting any money at all will be prepared to accept slightly less money to have someone else deal with the legal hassle of it all.