Silly REF question

[purplepandas]

I should know this but I was not involved in REF2014 due to a career break so I am a bit green about it. I am perplexed as to how you 'know' the status of your publications in terms of stars? Who judges them and how can you get an inking of where they might fit within the system. Asking as upcoming interview and I need to know and at least have an idea about my own work.

[purplepandas]

Thanks all and sorry to be slow to come back to this. It does seem rather arbitrary which makes me feel better about being a bit confused about how things are graded. I promise not to be an arrogant arse at all (truly) and was just thinking ahead really. I suspect it will be mentioned at interview hence asking.

And yes, the TEF, agh. Joyous.

Sunshine, tempting. You should fab to work with. I like honesty and transparency! I don't like arrogant arses. A win win.

[Rollinginthevalley]

And I've heard panel members say they've given 4 to articles and 2 to books.

[hayita]

Citations are also used as a proxy for quality in some departments I know. Yet the relevant panel last time gave relatively low rankings to some heavily cited papers and vice versa.

[MarasmeAbsolu]

Another trick I learnt is to make sure you submit to the right panel

... only if you have a choice in the matter - we do not, and get returned in "the UOA of Hell" because our management groups says so.

And yes, I am at a uni which still used journal IF as a quality proxy in spite of all recommendations.

[arclight]

Most departments will go through a process of internal and external evaluations. These can throw up very mixed results, so as previous posters have suggested it's probably best not to overhype your own papers if asked. Also, if you're work is interdisciplinary this can be even more difficult. I know people who were told they would not be included in the REF by their own department, only to go to a different department in the same institution to be told they would be delighted to include them.

Some people use journal
Impact factors as a proxy for a star rating, but this is not a great fit either. From talking to someone on a REF panel, publications that were in top tier journals could easily be given a 1 or 2** star rating while papers in mid-ranking journals could easily receive 4 star ratings. The panel member's view was that they really were trying to evaluate the quality of the research and not whether it appeared in a glam journal.

It's worth pointing out that nobody knows the exact form the next REF will take, so any evaluation at this point is not much more than a rough guess.

[MedSchoolRat]

It's really hard to say because so field specific. Obviously RCT doesn't matter in a humanities subject, but it's a high quality point in medicine. Another trick I learnt is to make sure you submit to the right panel. What one panel rates as high 2 another panel might give low 4 for. In 2014REF some panels systematically quality graded for each of the supposed criteria, and averaged for those items; other panels told their members to simply form an overall opinion with no breakdown. Who knows what will happen about consistency of approach in 2020REF.

Most panel members read ~600 papers. They get no pay for that, they can only give each paper 10-20 minutes. Write clearly to get best *s.

Journal isn't supposed to matter but actually, better journal = settles doubts. J. of No Hope = encourages doubts (in mind of panel member).

[Rollinginthevalley]

It's universities using the REF as a proxy for performance management. I value colleagues' opinions on which are my "best" pieces of work, but I think we're all second guessing blindfold in trying to grade publications. Heigh ho

[hayita]

I am perplexed as to how you 'know' the status of your publications in terms of stars?

In my area/department you are actually not told the grades your publications received at internal review. In REF0214 people were only told if they were going to be submitted or not.

I know internal grades from recent reviews but a neighbouring department seems to use wildly different grades than we do. From REF2014 results it's hard to work out whose internal grading is more reliable, so I would still struggle to benchmark.

[Sunshowercap]

Mainly, the REF is brutal crap. The TEF will be worse.

Institutional research reviews & gradings are demoralising second guessing of a panel 5 years hence for a REF for which we haven't yet received Panel guidelines or anything. They take up time & emotional energy, and they upset people. So think yourself lucky you haven't been involved.


And breathe.

[Sunshowercap]

We are recruiting at the moment ...

[purplepandas]

Thanks sunshowercap and I love the honesty of your reply. In particular I agree that it is best not to be an arrogant arse! Can I please come and work with you?

I will go and have a look at the REF2014 website again. I have also been considering TEF as I can imagine a question along those lines. It's an SL role. Best to be aware of all issues although TEF and REF pose different issues of course.

Thanks again for your help :)

[Sunshowercap]

I know of at least two universities, in addition to my own, where senior members of staff are reading likely REF2020 submissions, using REF2014 criteria. Go to the REF2014 website & have a look at the Publications section. You'll see the Panel Criteria document, check the descriptions of grades & criteria for your Panel & sub-Panel.

A word: don't give your own publications gradings, and actually - in a recruitment process, I'd prefer not to see people telling me on the selection committee what their institution graded each publication since 2014 at. I think an asterisk against possible/likely REF2020 submissions is enough.

Although in my first interview for a chair, I was asked to grade myself (I got the job - I gave sensible but modest gradings - I was told later I was too modest, but I think that's better than being an arrogant arse).