As long as you're not restricting direct access to their dropped kerb (if they end up forced into taking tighter angles for manoeuvring than is convenient for them is neither here nor there), you have every right to park there.
You may have unwritten/verbal friendly agreements with your neighbours, but you're under no obligation - and neither is the driver of any other road-legal car who happens to be passing/parking up.
If I'm at the front of a queue of 100 (otherwise local) people outside a remote Highland bakery 500 miles from my home, which has 100 loaves to sell for the day, I'm entirely at liberty to buy the lot every single day - and throw 99 of them straight in the bin - should I wish to, as long as the shop is happy to sell them all to me. It wouldn't be considerate, wise, cost-effective or in any way sensible, but I nevertheless am fully entitled to do so.
Parking on public roads is no different whatsoever. If it's a legal parking space then it's a legal parking space for ANYBODY with a road-legal vehicle. Even if you're a blue badge holder and the council reserve a disabled parking space on the road outside your house, it's reserved for ANY blue badge holder who wants to use it - not just you.
If he's just a bit thick and/or entitled, I'd speak to the neighbour about it and explain this. You could even act dim yourself and ask to see his deeds to check that the section of road is included as part of his own property.
If he still doesn't accept it, or is the intimidating bullying sort, I'd just tell him to raise his complaint with the police or the council and have them deal with me if I'm doing anything illegal and let them have a good laugh at his expense. If the latter, it may be worth leaving a motion-activated dash cam in the car, just in case of reprisals.