Whatever about the rights and wrongs of removing fence panels, he shouldn't be trespassing into your garden. Can you put a fence up on your side of the shared fence, a couple of inches inside your boundary? And then grow screening plants against it. It doesn't have to be a solid fence, post and wire put to shoulder height would do, enough to keep your DC and dog in and him out. If it's solid enough to support climbing plants then you won't even be able to see his drippy-painty fence.
If it were me I would be having a big argument followed by a complaint to the police if the trespassing continued, but if you have anxiety then just go for the solution that involves the least amount of confrontation.
How big is your garden? Could you afford to loose a bit? Could you put up a new fence? You could get your DH to use the space between the two, regularly so you don't loose rights to the space between but it would get over the fact that he can come into your garden.
Take lots of photos too, of the current fence with panels removed, of your neighbour standing in your garden (take the photos from inside if you're worried about him). Take photos while the new fence is being put up, put something like a ruler down to show the size of the gap between your fence and the shared fence. Keep the receipts for the fencing materials so you can show that you paid for it.
It might be a shared fence but that doesn't mean he's allowed to access your land without permission. Can you stick some nails in somehow so it can't slide up? Solicitors letter telling him not to trespass??
I would be tempted to plant a row of pyrocanthus along the fence with plenty of bonemeal for root growth whilst the soil is still warm and get it off to a good start for spring. Also a eucalyptus to grow fast and reduce the impact from the light.
If pyrocanthus are the evil bushes I have in my garden that grow fast, seem to be virtually impossible to kill, and have inch long hidden spikes growing all over (guess how I found this out) - then this....
I'd put up a second fence and, if you're not planning to move, ask a solicitor to write him a letter stating that he is not to touch it. Then I'd acquire a catapult and smash the bulbs in his security lighting plant some trees along the new boundary to block the lights from his security lighting.
They are indeed evilly spikey, but you can trim them into a nice neat shape against the fence, and they have pretty berries in the autumn which the birds like. If you put them in the right place, they will grow very well, and even if you cut them to the ground they grow back.