You both seem to misunderstand the definition of "agreement" in the context of the Act. It is an agreement between the surveyors, or, of the surveyors cannot agree, a third surveyor gets to decide. That is the specific legal process.
If you bothered to red the Act, you would see in section 10, it is provided, inter alia:
(10)The agreed surveyor or as the case may be the three surveyors or any two of them shall settle by award any matter—
(a)which is connected with any work to which this Act relates, and
(b)which is in dispute between the building owner and the adjoining owner.
(11)Either of the parties or either of the surveyors appointed by the parties may call upon the third surveyor selected in pursuance of this section to determine the disputed matters and he shall make the necessary award.
There is even provision for either of the parties to challenge the "award":
(16)The award shall be conclusive and shall not except as provided by this section be questioned in any court.
(17)Either of the parties to the dispute may, within the period of fourteen days beginning with the day on which an award made under this section is served on him, appeal to the county court against the award and the county court may—
(a)rescind the award or modify it in such manner as the court thinks fit; and
(b)make such order as to costs as the court thinks fit.
Therefore it does not require OP's consent - merely the consent of the surveyor appointed to represent OP's interests.
Oh, and it is a wall subject to the Party Wall Act - see s1(4).