Your neighbours will already know that they have been rumbled because the builders will have phoned them immediately after leaving your garden.
Somebody upthread suggested that the builders are probably related somehow to your neighbours, and I bet she's right.
I surmise that you live alone, OP? That's probably one of the reasons for your neighbours trying it on, sadly. They probably think that you, as a single woman, will be easily intimidated and won't make a fuss. Be sure to confound them!
I'd expect a visit this evening from your neighbours, who will be keen on either damage limitation or persuading you to let them go ahead with their wretched door. It might be an easier conversation on your own turf!
I had something similar before I was married, when I lived alone. I had two sets of CFNs.
The first one took down the fence dividing my detached house from his one Saturday. I came back from a day shopping at 7 pm to find the fence gone. I went to ask what had happened and his reply was, "I knocked to tell [note, not ask!] you earlier, but you weren't in". Bollocks. Apparently he did it "because he couldn't get his lawnmower between the fence and the house".
A month later, I came home from holiday to find a massive fuck off conservatory built a metre over the boundary! I joyfully reported it to the council.
The other set were downhill from me, but we had a shared garden wall which collapsed one day. The wall belonged to the neighbours but they came round and said that they wanted £10k (!!) for a new wall and repairs to the garden because it was apparently my fault for owning the land uphill from them. I did a bit of research and discovered that he had replaced a fence with a breeze block wall with no footings ten years previously. I advised him to claim on his insurance, then he admitted that he didn't have any because he'd been convicted of handling stolen goods! Cheeky bastard. He got nowhere with me.
Both CFNs thought they could get away with doing what they wanted because I was a young woman living alone.
In summary, as others have said, stand up to the cheeky fuckers!