@Tarr
Oooh, I'm on the fence here. (See what I did there!) 
On the one hand, ivy IS quite strong and a bit destructive, and you should have kept it in check, but on the other hand, he should have said something to you about it, and it is his fence.
We had a big raucous, ugly, low-lying fern, (belonging to our neighbour,) that was putting a strain on the 3 ft high wooden fence between the 2 front gardens. (And the fence was our responsibility.) The fern was also creeping through the slats in it etc, and was 8 foot in circumference (6 foot on his side and 2 on ours.) So it was affecting our lawn.
So, we went next door and said what was happening, and asked the man (who was widowed and lived alone and was OK but very much kept himself to himself) if we could cut the big low-lying fern back. He said 'knock yourself out love. D'ya wanna cut my side too, coz my lawn's turning to shit under it too!'

So we cut it back 80%, and we were all happy!
So I do think he should have said something to you if it was affecting the fence!
And I don't think he has a case for asking you for the money. If I could afford it, I may offer, say, a third towards it. But that doesn't mean you should. You could just say 'it's your fence, and you never said anything about the ivy before...' (He hasn't has has he?)
Because that may change things a bit.