Neither of you are 'on the back foot'.
He's clearly got under your skin, talking about whose house is diminished in value because of the access situation. In truth, neither are. He's chatting shit, and is doing so because he thinks it will help him buy your house for a diminished value, or maybe because he just likes chatting shit. Discard. Ignore.
The access issue, however, is worth trying to clarify.
If the wording in your and his title registration is ' to pass and repass' - commonly used in these situations, he has no right to park.
You could check out the wording with a solicitor, to be completely clear on where you stand. I would not be asking the solicitor to write or take action, because 1. expensive and 2. not a good way to sort neighbour problems.
I would take anything on Garden Law with a shovel of salt - plenty of barrack room lawyers there, who may well give wrong info.
Don't take notice of what he says.
If someone parks on your land, ask them to remove the car. If you can arrange it so that them have to move for you to get your own car out, you'll invent a need to go out in your car every 20 mins, until they get the message.
Don't argue with anyone. If needed, send a written letter to the owner and also the occupier, quoting what's in your registration, and why they are not allowed to park..Be clear, brief, and polite.
Still problem,( but there almost certainly won't be), another letter, saying you have received legal advice, but hope to resolve this without legal escalation, and restate your position.
Always keep copies of letters. Always stick to one point, and write as if it might be read out in court-martial clear, polite, firm.
Nearly everyone who threatens others is a bully. Bullies back down if they don't get a 'victim' response.