OP YANBU, but equally it's not the be all and end all.
I'm the first person in my family in three generations to go to a non-Oxbridge university. I applied to keep people quiet, but had no intention of going because I know it wouldn't have suited me.
I think my sister hit the nail on the head describing her experience - you take the people who have always been the top of their subject at school and suddenly they aren't the best, and they are surrounded by people who seem cleverer, happier and more confident. The people who realise that earlier on do well, the people who take longer to realise it begin to doubt themselves and struggle.
One of my friends went and hated it, and my sister told them that as long as everytime you get upset you tell yourself "it's just a means to an end" then that's fine. Frankly I think "just a means to an end" is a pretty miserable way to get through uni, but I'm feeling that at mine too.
My personal belief is that you never know if a university was right for you until it's too late. I would have been happier doing a different course somewhere else, but I'm determined to graduate. The pressure has only come from me, but I feel that at Oxbridge I would have become too competitive.