You are preaching to the converted LRD! I spent three years after my PhD as a very junior academic at a small, old-fashioned, conservative Oxford college. I was regularly the only woman on high table in the evenings. I felt constantly self-conscious and on display as a young women in that environment. Needless to say I had no colleagues who were seriously disabled, or from an ethnic minority. I was made to feel self-conscious about being RC (as opposed to CofE) for goodness sake! There were at that time two women (only) who were full 'fellows' (senior to me), out of about 40, but neither of them came in often. All the other women were on temporary contracts of one sort or another, or were administrative staff. I am used to very male, traditional institutions after years of them but I was shocked by this experience, which happened very recently by the way.
The casual, unthinking prejudice about practically everything from men who were perfectly nice in many ways but just horribly complacent drove me completely crazy; and above all the unproffessionalism and the closing of ranks. I also found it, intellectually, fairly mediocre actually.
I have had better experiences in other contexts within both Oxford and Cambridge (I've both worked and studied at both) - especially Cambridge which is on average more professional and forward looking I think (probably partly because departments have more power relative to colleges). But I appreciate that precisely because colleges are all independent institutions everyone has very different experiences.
I got a lot out of my own undergraduate experience (in Oxford) in many ways but I wasn't very happy there and I really don't think Oxbridge is the be-all and end-all academically or in any other way. Lots of people get a better education elsewhere.
As an undergraduate I think most female students - as I was certainly - are reassured by the good gender balance among undergraduates and don't really notice the disparities among the staff. But I can imagine that a black sixth-former visiting Oxford or Cambridge would feel as I did on my first day as a young, female, fellow. I am an outgoing person and I don't mind being the centre of attention some of the time so I made the best of it, but it wasn't comfortable or a good working environment and I wouldn't have chosen it. A man who has spent all day every day at work surrounded by men of a similar background to himself simply has no idea how it feels to feel self-conscious all the time every day. To be honest, I didn't fully realise how uncomfortable it was until after I had left.