TEF results - what value to attach?

[BachingMad]

I have been looking at these results today in the media and wondering what importance to attach to them, given that places like LSE and Southampton, well respected RG unis, have come out so badly.

I am aware that the testing methods are a bit untried and untested, but also that some prestigious unis seem to rest on their laurels. Also, the previous ratings, by research quality, does suggest that some lecturers will be more keen on furthering their research than teaching students, which is what the students are paying increasingly large sums/running up debts to pay for.

This will not affect me until next year, and obviously the results have come too late to affect decisions for this year. However, in principle the basis of the testing seems sensible, but how much value to attach?

As an academic who is extremely familiar with five Russell Group institutions (3 gold, 1 silver and 1 bronze), I would treat these results with extreme caution.

I do not believe that LSE/Southampton teaching is significantly different from teaching in institutions such as Imperial/Nottingham which obtained gold.

The assessment procedure is fundamentally flawed, with the criteria being such that it is much easier for low tariff institutions to achieve higher teaching scores than high tariff institutions. RG institutions that had very similar quantitative data were separated into gold, silver and bronze on the basis of 15 page statements. This separation was highly subjective and probably politicised.

Also, the previous ratings, by research quality, does suggest that some lecturers will be more keen on furthering their research than teaching students.

But this simply cannot explain the variations in awards between very similar institutions. My own experience (as an academic who is involved in monitoring teaching at other universities via external examining) is that several institutions that got gold actually have significantly worse teaching in my subject than some which are lower ranked. TEF is of course at university level, not departmental level, but I don't believe that the separation between institutions with very similar students/courses is meaningful.

I'd be cautious too.

A big issue that a lot of people are concerned about is the fact that students tend to give worse evaluations to women teachers and teachers who are not white. There have been a huge number of studies looking at this. For example, if students think they are being marked by Dr Smith, a white man, they are more likely to think his feedback is fair, objective or authoritative. If they think they are being marked by Dr Smith, a black woman, they are more likely to think she's being biased, unfair, or incompetent. Even if the actual feedback is identical, the assessments from students won't be.

Similarly, students expect different things from male and female staff. Men are usually forgiven for being more hands-off and research-orientated, and these things are seen as professional. Women are expected to spend more time doing pastoral work (even if that's not their job).

I'm not saying this to be rude to students - we all have the same biases. But it's enough to make me think the results will be hard to interpret.

Dragon, I am shocked that there would be the prejudice which you refer to. Do you have links to any of the studies? Do some unis have more diversity in their intake of lecturers? I would have assumed that the levels were broadly similar.

I don't think gender/ethnicity biases lead to the variations between institutions, as the diversity levels are broadly similar between comparable institutions. ("Top" institutions will often have more white men though.)

However, the use of measure such as NSS scores which are inherently flawed (not just through student biases) to measure teaching quality, rather than expert assessments and observations of teaching, does undermine confidence in TEF as a whole.

My understanding is that LSE students take a certain pride in the fact that their institution always comes bottom of student satisfaction surveys. Whether this attitude influences future surveys is anyone's guess.

DS messaged me this morning with the impressive news that LSE had been ranked below University of the Arts Bournemouth, indicating a satisfaction that their reputation was intact.

I would not advocate anyone going to a top London University unless they were a self starter and wanting to fully engage in their subject. But equally a number of DCs contemporaries seem to be really thriving at Imperial/LSE/UCL because it is very unlike school and, though there are impressive resources available , they are expected to take responsibility for their own learning. Horses for courses?

There are 9 who have just completed DS' course at LSE. 3 will go into employment, 1 is heading for an Ivy League post grad course (with funding - and bizarrely has worked out that he could complete this Masters simply by repeating content he has covered in his undergraduate degree) and the remaining 5 will stay on at LSE. They will all have received offers from elsewhere, suggesting LSE teaching at least in DS' subject, cannot be that bad.

They will all have received offers from elsewhere, suggesting LSE teaching at least in DS' subject, cannot be that bad.

But they came in with very high grades, suggesting they would have done well anywhere, regardless of the teaching quality.

I agree, however, that it is very unlikely that the LSE teaching in Economics would be worse than that in places which got gold.

Why shocked?

Surely you recognise that people in the world are still sexist and racist on occasion - and often it's not active bias but internalised assumptions.

I'll look out some studies, though frankly, having read them a long time ago, all I'll likely do is google then check the reliability, which you could probably do just fine.

Yes, different universities have differing levels of diversity. Some (naming no names) are pretty awful; others much better. Until a couple of years ago, it was perfectly possible to do, say, an English degree at a very good university that employed no women as professors, and no black or ethnic minority staff in roles of reader or upwards.

Yes, they may have done well anyway regardless of teaching, but if there was a general dissatisfaction with the teaching at least some of them might have moved elsewhere for their Masters degree. It's not as if the LSE is pricing competively.

That said one course and all the usual caveats apply.

I would not advise any applicant to pay attention to the TEF.
Despite being callled the Teaching Excellence Framework, teaching excellence is not measured in the TEF. They basically look at existing metrics (NSS, employment destinations, retention) benchmarked against what should be expected (to deal with differentials in the type of institution) and then contextualised through a statement provided by the institution. So this really teaches us nothing that the perusal of leagues tables can' tell us. All Bronze universities will have staff who teach brilliantly and all Gold universities will have staff who don't teach that brilliantly - both in terms of differentiation between departments and in terms of individuals. The difference lies in university policies, strategies and approaches.

Plus, Bronze, within the terms of reference of the TEF, is still good. Bronze is higher than the baseline standard of the Quality Assurance Agency. Bronze should not be treated as the equivalent of, say, a requires improvement in an OFSTED. And if the NSS and the REF are anything to go by, universities that gained Bronze and Silver will be working very hard over the next years to improve, so they may actually be quite good bets for good student experience in the future.

Have a look at the individual results for universities on the HEFCE website at www.hefce.ac.uk. If anything, this ought to give the lie to the idea that universities, and university staff, don't care about teaching. We do. It is just that, unlike at school, teaching is not all that we do - academic staff are not full time teachers.

(In the interest of full disclosure, I am an academic at an institution that has been awarded a Gold).

So this really teaches us nothing that the perusal of leagues tables can't tell us.

I think it's actually worse than this. The three categories will be taken to mean that Gold > Silver > Bronze although in reality high tariff universities were held to higher standards to get Bronze than low tariff universities were to get Gold on criteria such as student retention etc.

The difference lies in university policies, strategies and approaches.

I am far from convinced about this. I am an external examiner at two places, one of which got Bronze and one got Gold. They seem very, very similar both at departmental and university level.

The three categories will be taken to mean that Gold > Silver > Bronze although in reality high tariff universities were held to higher standards to get Bronze than low tariff universities were to get Gold on criteria such as student retention etc.

True, but that's the impact of benchmarking. An institution with, say, a high proportion of mature students shouldn't be held to the same standards re retention as an institution with a much lower proportion, because mature students are more likely to drop out due to Life etc. The problem lies in assuming that a statistic about retention can tell an applicant anything useful about their personal chances of staying the course.

I am an external examiner at two places, one of which got Bronze and one got Gold. They seem very, very similar both at departmental and university level

I'm thinking more about the things that are included in the provider submissions - I found this quite helpful in making sense of it all. wonkhe.com/blogs/tef-results-what-the-panel-statements-say-and-dont-say/ . I suspect that an awful lot has depended on the way in which the submissions were written and that has to do with mindsets in the higher echelons, not what happens on the ground.

A 15 page submission should not be used to make such a radical splitting - one which will likely have huge impacts on student recruitment and (potentially/probably) fee income.

Parents/students will choose Gold over Bronze, without understanding what the metrics actually were. I think it is very unlikely that people will know that the benchmarks required to achieve awards varied so much between institutions.

.

I think we are in violent agreement....

OP, if your interest is where you can find excellent teaching, there are basically two things you need to be aware of:

*the TEF did not at any point actually examine actual teaching, e.g. through sitting in on classes or evaluating teaching resources: this is not some kind of Ofsted (the closest it comes is student satisfaction, which was only one of the criteria)

*due to the benchmarking universities were graded differently for the same results: so an institution that got Bronze may actually have scored higher than one that got Silver. If you want to choose the one that scored higher, you need to look at the actual scores.

It is also worth considering that scoring highly in some of the categories may not mean what it looks like: drop-out rates could be affected by things like high living costs (explaining why London did unexpectedly badly), employment rates after 6 months would punish institutions where many students carry on to do further study or which are based in areas of high unemployment, student satisfaction could partly depend on other factors than actual policy (as per LRD's post).

What everyone said (although every expected London to do badly)!

Happy with our silver

actually yes, titchy, you're right about London, don't know why I put unexpectedly

Thanks cory, that's helpful. I am actually thinking of applying to Kent, so pleased it came out well, even though I will take it all with a pinch of salt.

For a comparison with the actual metrics (benchmarked so should take into account the different student profiles each institution has) have a look here:

WonkHE

The silvers at the bottom of the table are interesting....

The assessment procedure is fundamentally flawed, with the criteria being such that it is much easier for low tariff institutions to achieve higher teaching scores than high tariff institutions. RG institutions that had very similar quantitative data were separated into gold, silver and bronze on the basis of 15 page statements. This separation was highly subjective and probably politicised.

Indeed.

The TEF rankings were produced by institutional-level data. So it's much easier, for example, for an ex-Poly with low entry scores, but giving quite a few 2, i degrees, to have very high "Value Added" scores, than for any of the 3 Russell Group universities I've worked at, which require high entry level scores, and then evaluate students to much tougher standards. I've been External expert at quite a few post-92 universities, and the standard of teaching & research is generally lower ... the staff have to teach many more students over many more hours, for a start.

The TEF has NOTHING to do with the everyday teaching of undergraduates & postgraduates. There is NOTHING in the TEF that will tell you anything about the day-to-day experience of your DC, nor the "quality" of the teaching they receive week in, week out.

And how would you judge that quality anyway? We often think that student evaluations of teaching tell us something. But ...

If I push & challenge, if I question, and require students to do the hard thinking, I may often get lower "satisfaction" scores. I will probably get lower satisfaction scores because I'm female, anyway.

But my teaching is likely to be better for the students' learning than someone who goes a bit easier on the students. My emphasis is on students' learning - I try to facilitate, provoke, and extend their learning, not "teach" them.

But that approach does not make for easily satisfied students ...

There's actually a fair bit of research emerging which shows that student evaluations are better at predicting students' sexist & racist biases, and often the students with tutors getting high "satisfaction" scores, don't do as well as the students of tutors with lower student evaluation scores.

BTW, my institution was ranked gold, so none of this is sour grapes - the TEF is even more ridiculous than the REF.

At least the REF involves peer-review.