Advice on how to deal with scaffolding situation

[Turniptracker]

Long, sorry! I'd like some advice on how to deal with a situation with next door and their builders. I am not very assertive and often just come off aggressive when challenged so I'd like some advice on the most calm and professional way to deal with this.
Next door are having a loft conversion. It's been going on about 4 weeks and last week some additional scaffolding was added. I thought nothing of it. Yesterday I realised they are using my outside house wall as a support for the scaffold. No one asked me and the pole is considerably over my land boundary. It's an old house and I don't want it to be damaged.
I went round to my neighbour and explained I had just noticed my house is being used as a support and I was not asked so could they please ask their builders to remove it. They said they would speak to them tomorrow, they hadn't realised.
My issue is this. It is the sheer audacity of the builders to just do what they want without asking me first. This makes me automatically want to refuse any request they make in hindsight to use my wall. They just think they can do what they want. They are legally required to ask my permission and they haven't. If they say they absolutely have to do it for safety reasons how do I handle it? I don't want to hold up the process of finishing the conversion but I don't want my house damaged and it's the principle they didn't even ask. Any ideas? I still have to live with my neighbour after this despite the fact she has always been a selfish nightmare to live next to and cut down part of my old oak tree without asking the other day too. Sick of being treated like a doormat.

I'm sorry I don't know the correct process but I'd be out there quick smart and take plenty of photos now to ensure you have proof of anything does happen.

Scaffolders are amazing at putting up what they need in tight spaces. I doubt the have to use your side. They are probably cost cutting or lazy.

I wouldn't beat around the bush. I would tell them they are trespassing and to remove it from your property by the end of the day.

It's one thing to ask permission first and show you prof of their third party insurance, another to overhang air space and quite another so actually use your property for support!!

Time and again there are threads on here where people are the nice neighbour and then end up with damaged drives or property which they can't get rectified.

Is there a party wall agreement and do you have a surveyor?

I don't share a party wall with them. There is a decent gap between our houses. I basically don't want to have to claim against them and go through all that hassle. Would rather they just removed. They've already dropped debris on our drive including a plywood board for height which smashed on our drive and would've seriously injured someone and nails/screws

In honesty this is rarely an issue that causes damage but completely appreciate your feeling about it and they should have asked. Hopefully it's not a pole in contact with your wall as that will do some limited damage, albeit you might not even notice and a block of wood should prevent it.

Keep a close eye on debris, often small sharp objects (screws, nails etc) are dropped by workers without them noticing

It is pressed against my wall and being used as a support. It's not overhanging air space I wouldn't care about that it's the fact it is braced against my wall and no one even asked permission. Even typing about it makes me angry

Hopefully it will get removed soon with an apology from the scaffolder! Try not to get into a disputed with your neighbour as they might not have seen it.