Neighbours scaffolding drilled into my house

[52andblue]

My neighbours scaffolders arrived Christmas Eve at 8am. Very noisy. I went out to move my car as they were chucking stuff around. I spoke to them as the scaffolding was being erected in the gap between our houses (about 5ft). I said I didn't want any scaffolding on my house. Later they drilled into the sandstone of my house and put in a steel hook which they have attached some scaffolding to. They'd gone before I saw it. Neighbours nowhere to be seen, presumed away (not answering door).
Today I see them. Pleasant hellos all around. I ask when scaffolding coming down- they don't know & comment how noisy it was for them and it becomes clear they've been home all along. I say I'm not happy with the hook which will leave a hole and they tell me to 'take it up with the scaffolders'. I say that's not MY place and I'd like it sorted. No reassurance or apology forthcoming.
AIBU to take a pic, and write to the neighbours rather than the scaffolders? Neighbour is older but does DIY - quite happy for him to fill hole but would like it sorted out soon.

I'll try to attach some pics

You need to contract your house insurers straight away. Any conversation you have with either the scaffolders or neighbours will be hearsay otherwise
Keep everything above board and take lots of photographs and record the dates you tried to get hold of them
Refer them back to your insurers every time

Pictures, send to neighbours and the scaffolding company as a compliant with a request for a response from their lawyers.

I can't believe you've left it this long!

Have you managed to get it taken down yet, OP?

Not yet no. No reply from Scaffolders (on hols till Mon I suspect)
No reply from neighbours when I knocked (don't have their name or number). I will want to sell at some point & don't want a dispute with them as it's a tiny village and they could make my life very difficult if they chose to. I would be content with them filling the hole for me (I'm physically disabled and cannot do it myself whereas he is a DIY guy) but I wonder if they will even do that considering that they were there all along, were very dismissive when I raised it (in front of kids). A neighbour came out when I was taking some pics for evidence 3 days ago and asked me what I was doing. That shook me up a bit.