If you can't avoid entirely, a stock reply would be best:
'You really shouldn't make it quite so obvious that you're not happy with the way your life has turned out.'
'You sound so bitter. I'm not sure that having a go at me is going to make it feel any better, though.'
'Still bitter at life? I'm not sure I can say anything that will help you feel better, sorry.'
You don't need a lead-in to make these replies. The sad truth is that he attacks you because he hopes it will make him feel better, because whatever it was he wanted hasn't materialised, and he hates that you're clearly happier than him. So when he says something goady, don't reply to the words coming out of his mouth - reply to the sentiment behind them. It will hit home, even if he just insults you back...
eg:
Him: 'Who gives a shit what you say - you're utterly stupid, always were...'
You: 'You sound so bitter. I'm not sure that having a go at me is going to make it feel any better, though.'
Him: 'You fucking loser - I heard from DS that you were out again on Saturday, ever going to give up on trying to get another man? Nobody wants you...'
You: 'Still bitter at life? I'm not sure I can say anything that will help you feel better, sorry.'
Every time, reply along these lines. They will hit where it hurts, because it's the truth.
Sad man - looks like he lost the gamble.