Noddyholder, I have to go the small claims court route from time to time to get my money when people don't pay my invoices (I'm self-employed). It's pretty straightforward: you write a formal letter stating your case and saying that if you do not receive payment within fourteen days you will take legal action. Send it recorded delivery so you have proof they've received it. Most people pay at this point.
If that doesn't bring any joy you file a court claim. It's v. easy you can do it online. You have to pay a small fee, which is a percentage of the amount you're claiming for for a £500 claim the fee would be under £50 I think, and gets added to the amount of your claim, so you get it back if you win. Keep copies of all invoices and correspondence, and make a note of time, date and content of all conversations. If you have any further conversations, consider using a recording device (even a cheap dictaphone) to record it.
I'm happy to help you if you do want to go the Small Claims route, as it's easy when you've done it before and intimidating if you haven't. I can write you a specimen letter, and talk you through the online claim.
If he's actually threatened you, that is a criminal matter and you should go to the police.
CAT me if I can be of any help. You shouldn't feel you've lost the money, as he has no right to it if he hasn't done any work.