There are two issues here (and your neighbour bears the burden of sorting out both of them).
Party Wall Agreement. Yes there is a statutory obligation on neighbour to serve a notice. Once you agree that the works may proceed by countersigning, that becomes the pW agreement. If you dispute it then you serve a counter notice (these are forms you can just print off the internet). Both parties appoint a surveyor (the building neighbour pays the cost of both surveyors) who together draw up a more detailed agreement of the scope of works including access arrangements and a schedule of pre work condition. This becomes the Party Wall Award and if it is breached/damage caused then it is also the mechanism by which the surveyors agree repair works which must also be funded by neighbour. It is a process that is intended to keep disputes out of court though if your neighbour fails to follow it you can apply to court to enforce it.
The purpose of it (I can see posters have said they can't see the point) is that it gives you a direct remedy against your neighbour. If the work proceeds without a PW agreement and your property is damaged, You can sue but you have to sue the tortfeasor who is not, as you may think, your neighbour but the builder who does the physical work. You will also have a claim for breach of statutory duty against your neighbour though.
The PW process doesn't need to be "costly" . As I said, the prescribed forms are available on the internet. It can start adding up if you dispute the scope of work set out on the notice (you can't refuse the works outright but can object to how they are going to be done or insist on more info) or you want a formal schedule of pre condition agreed by surveyors, meaning a PW award has to be drawn up. (When we served our neighbour with a PW notice they were willing to forego a schedule and instead emailed me a series of photos of their property as proof of its current undamaged condition).
The other aspect is the need for access and placing of scaffolding on your property. Your neighbour has NO right to this except for essential repairs (Access to Neighbouring Land Act). If your neighbour comes into or places material on or even oversailing your land, without permission, he (or rather the tortfeasor) is committing the tort of trespass and will owe you damages for the trespass and any damage caused.
You can come to an agreement by granting a licence for access. If you feel benevolent you can waive a licence few or state the sum you want depending how long the scaffolding/access is required and how far the intrusion will inconvenience you/spoil the enjoyment of your property (it's a good idea to make the licence fee weekly/monthly in case the work overruns though you can also specify an end point in the licence after which you will be entitled to damages for trespass at a higher rate). There are no prescribed fees, You can really demand what you want but may want to bear in mind neighbourly relations and whether you might want a similar favour in the future. (Our scaffolding over sailed our neighbours garden slightly but they waived the right to insist on a licence fee as they might have wanted the same from us in due course.) If you refuse any access then your neighbour has to find another way such as having materials craned in over his house. If trespass occurs without permission (i e. Workmen jump over the wall and start erecting scaffolding on your patio) you can seek an injunction from court.
The licence issue can be dealt with separately or wrapped up in a Party Wall Award (see above).
It sounds like access via your house is essential, which is good as your neighbour can't just go ahead. But as posters have said, make clear asap that you aren't consenting to what he has proposed and that you do not give permission for access and point out that he is obliged by law to serve a PW notice.
Hope this helps.
I am a solicitor specialising in property damage by the way!