I second asking on the GardenLaw website.
Did they happen to mention how long ago the previous resident allegedly 'gave them permission' to have some of the garden. How old is the fence (or how old does it look)
Do you have any photos of the house from previous years. Aerial photos can be very useful?
Do you have legal insurance?
How detailed are your deeds? Do you have any measurement on the plans. If you have the original planning or builders plans then they may have measurements.
I'd see what the GardenLaw people say but I would write to him saying that you are putting the new fence on the boundary and I'd give him a period of time to reclaim his plants. I can't see that there is any point getting into discussions with him.
I would phrase any letter as simply as possible. Maybe you could include a plan
Dear Nieghbour, just to let you know we are erecting a new fence along our boundary on XX date. I understand you have some plants in our garden. If you want them can you please can you remove them by the xx
I'd make it a very sturdy fence and as high as possible possibly electrified too
Take lots and lots of photos from every angle before and after erecting the fence.
Btw - I'm not a lawyer or anything similar.