What have you agreed about doing the work and to what timescale? Whatever it is, it is clear that he is messing you around and not turning up and getting on with the work, as would be a reasonable expectation of a paying customer.
And he has some of your money which is even worse, as you have kept your side of the bargain by paying it upfront and he hasn't kept his, by turning up and getting on with the job - a breach of contract.
You need to read what is in the contract but the contract itself must be reasonable and can't have unfair terms. It can't absolve him of liability to keep his side of the bargain and act fairly towards the customer.
You have rights under the Consumer Rights Act 2015:
[From the CAB website]
"There are now also new, clear rules for what should happen if a service is not provided with reasonable care and skill or as agreed. For example, the business that provided the service must bring it into line with what was agreed with the customer or, if this is not practical, must give some money back."
Here is a link to CAB with phone numbers if you want help:
https://www.citizensadvice.org.uk/about-us/information/the-consumer-rights-act-2015/
However you need to confront him and not be afraid, telling him your reasonable expectation that he turn up and do the work as agreed (in 6 days) and that as he hasn't done so, at both great inconvenience and substantial cost to you, paying a deposit in good faith and against the promise to do 6 days' work) you are terminating the agreement, paying him for what little he has done and require the deposit money back (minus the payment for the little work) and you will contact local Trading Standards if he refuses.