Um, he's an idiot (putting it mildly). And what kind of an extension is he having that needs scaffolding for six weeks?!!
Option a: being nice. Tell him he can have access for max. 2 weeks for foundations, wall building and any edge finishing round roof. All the rest can be done from the top/rear.
Option b: your legal right, should you choose to exercise it. If he's building on the boundary, and for any construction up to 3m from your boundary, you can refuse him entry and request a party wall agreement. This will take a few weeks to set up. He will have to pay for it all, including the same or a separate surveyor of your choice. The purpose will be to check the foundations don't undermine your side of the boundary, no damage to your property, and that eg correct working hours are maintained, permission is sought for coming on your side and they only come on if you agree, etc. It would delay his project easily by 4-6 weeks, cost him about £1500 or more, and royally fuck him off, no doubt.
So as he's your neighbour, I'd aim for option a, but strengthen that request with a glancing reference to option b.
Eg, 'Ah just been thinking about what you said ... TBH it will be a bit much having your builders and the scaffolding here for 6 weeks, so I think 2 or so should be ok? I don't want to go down the route of insisting on a party wall agreement, as I know that would hold you up and cost you thousands, so can we agree that they'll have access to build the wall and finish edge of the roof/guttering if necessary, but it really shouldn't have scaffold up for 6 weeks? Also I need to make sure my patio isn't damaged at all.
He's going to be cross if you disagree in any way, but just telling you he's putting it up is bullish - he thinks you don't know about your party wall rights, and will be scared of him.