It was definitely worth it for me. I was at King's, Cambridge, in the mid 90s. I come from a state school and deliberately chose a college which had a high state school and female ratio and was known to be politically left wing and less stuffy/formal than many colleges.
It is definitely worth choosing colleges carefully as they are so different. Eg in my year, at Peterhouse there was only 2 girls in the year. Both dropped out. Magdalene was very macho/sporty/public school from what I saw.
I loved my time at King's, though with hindsight I wish I'd taken more advantage of the world class teaching and drunk less! Still managed to come out with a 2:1 though.
I am still great friends with all my King's friends now, 20 years after we graduated. It was so lovely, after years of being bullied or isolated at school for being too "swotty", to be surrounded by others who were as bright and motivated as I was! Socially, it was the making of me.
Intellectually, I did find it difficult to be average after years of being the best, and for a long time I lost confidence and thought I'd got in by "accident" and everyone else was brighter than me. It took me a while to work out this wasn't true.
The supervision system can be quite intimidating for introverts and the system does rely on you being motivated enough to do your own work.
I've felt having Cambridge on my cv has been helpful, but I'm in law where it's very academic, hierarchical and traditional. When I see oxbridge on CVs, I consider it a good starting point and it would help them get an interview but not the job.