These neighbours recently bought the property. Of course they know it's listed.
To buy a listed house and think they can do this they're either
a) actually, genuinely learning impaired. They would have had to listen to the estate agents and their solicitors telling them about it being listed and there being restrictions. Then most reputable builders would look at it and refuse to do the work without evidence that they had permission. It's hard to 'accidentally' not hear all of that information without a genuine problem.
OR
b) criminally self-absorbed. Because they have committed an offense. Given the speed with which the builder removed his stickers (and how quickly they attempted to do the work) I would expect this to be the truth here.
Do you know your neighbour's name? Can you find out if they are related to the builder who did this?
OP - as entertaining as this is for us, I feel really sorry for you. It must be really stressful. And yes, you may need to consider if they've damaged your property by removing the brickwork. Which is just a whole other layer of shit to deal with. Probably they haven't, but do consider whether they've compromised the integrity of the wall.