"...the down pipe is 100% on his property,"
In that case the down pipe belongs entirely to him and it's up to him to sort any repairs out.
Given that these are likely quite old properties then you will likely have gained the prescriptive right to discharge the rainwater onto his property. So he can't do anything to stop the water from your gutters going into the down pipe.
.
"...the shared gutter is asbestos"
I'm not too sure what you mean here, but any guttering that is on your property is your responsibility and any guttering on his property is his responsibility.
I imagine a situation where the whole guttering at the rear of the two properties is asbestos as well as the downpipe.
In this situation, I would suggest that you first consult a specialist company that deals with asbestos. Depending on what the guttering is like there may be no need to remove it at the moment (certainly not immediately).
When it comes to who pays. You are responsible for any work that you choose to do on your own property, he cannot force you to do any work on your own property. So, if you choose to do any work on your guttering then you will pay for that.
He is responsible for paying for any work done on his own guttering and the down pipe.