Oh I am so angry for you!! 
I agree with PPs- take photos of the work today to show the mess & that the work is new.
Tell the builders they are trespassing. They need to stop the work immediately.
Don't give an inch when you speak to the neighbours. Allowing them to keep their doors devalues your property- it is the same as handing thousands of pounds to them!
And please keep us updated! I want to hear what the cheeky buggers have to say.