He's only 17, so there's still time to turn things around. Stephen Fry was in prison at that age and yet ended up at Cambridge!
Don't push him too hard too fast. Personally, I would get him on some kind of medication. People knock them, but anti-depressants, benzos, etc, really can be life savers. If you could get him on the right medication, he'd find social interaction easier (I believe seroxat and klonopin are recommended for social anxiety).
Next step would be to get him some therapy. He needs group therapy, not one to one. He needs to be in some kind of support group, with other young people who suffer from anxiety, depression, bullying, etc.
After that, a part-time job. Maybe a village shop – something easy and low-stress. It will give him an enormous boost. The awful thing about school is that it's your whole world. When you're in your 30s, you know bullying is not normal. You understand that these are just arseholes being arseholes and that it's nothing to do with you. When you're 14 or 15, however, you have no life experience and so you take their spite personally.
He just needs to get out in the world and mix with people. Once he does that, he'll realize there is a vast range of people out there, and no such thing as normal. You can do A-levels and go to university at 19 or 20 (or 25 or 40). I'm 46 and re-training for a new career. His life hasn't even started yet.
I really hope things work out. x