I'll try and condense this as much as possible and try not to drop feed!
We started an extension 6 months ago. Project was supposed to be 12 weeks, we are now in week 25. We stupidly don't have a contract with the guy as we knew him, and clearly naively, thought we could trust him. We didn't have a defined payment schedule (yes I know!!) And have paid him throughout the project whenever he has asked/invoiced us. We have copies of original quotes, invoices and all communications (albeit some are text messages but still a trail). We have ended up paying more than the original quote and are/were fully paid up. We had a conversation with him where he told us we only had to pay him the cost of the work to the upstairs of our house, which has not started yet and that everything else was paid. We have a spreadsheet and have tracked all costs according to him, plus our payments. There is a clear discrepancy i.e. we have paid more than we should have and he is unable to explain what the extra amount has covered.
We are left with an almost finished extension, but one that still has window panes missing, plumbing to be done, electrics to be fixed, as well as things like needing fire doors installed. For the past 3 months we have had him turn up for an hour or two once or twice a week, and progress is just not being made.
We have today spoken with the plumber, who is being paid directly by the Builder, and he was told today by the Builder that we are going to be paying him the remainder directly. Clearly we are not as we have already paid the builder this money. He was also told that we still owe him (the builder) money.
So we are basically stuck - he has us over a barrel and it has gotten to the point where we cannot get any resolution through pleading/negotiating/whatever with the guy. We need the project finished, we need it signed off by building control, yet the guy and the trades are not finishing it (we can understand the trades if he hasn't paid them) and we have paid more than originally quoted. I have the building control inspector coming on wednesday to discuss what still needs sign off.
Does anyone have any advice as to what we can do? I need to speak to trading services, but I'm worried we don't have any recourse as we don't have a contract.
And please no lectures 
, we know how stupidly naive we have been!!!