In Defence of REF

I never thought I would write a post with this title and, after last week, it is even more surprising. Nonetheless, I am a modest fan of REF and this post sets out why.

First, what might be construed as my (many) conflicts of interest.
I am partly responsible for our Faculty’s submission to one of the Research Excellence Framework (REF) panels. Last week science was my displacement activity, as I collated and moderated the results of our last REF reading exercise. Eight papers chosen by each member of staff in consultation with their head of Department, each read by two professors (I was one of these too). Where grading or its justification was contradictory, a third reader stepped into the breach. After moderating the choice of the best four out of the eight, I moderated the likelihood of a member of staff being returned or not. I estimate a good 30 h just for the moderation and collation. Maybe I take it too seriously, but in essence that is what it means (or should do in my mind) to be a professor. Your research at times seems like a bit of a hobby, as you help deliver the goods for the institution.

The concerns regarding how panels will deal with the mass of papers have been voiced repeatedly. Most recently in Philip Moriarty’s post at IOP
and at Telescoper’s blog. I agree with both analyses.

We are reassured by HEFCE, RCUK and panel chairs that citation data will either not be used or be used with a light touch. Light touch is presumably to push a paper up, not down? So they will be reading the papers. Can they do this? Telescoper’s post clearly demonstrates that panel members cannot read the papers according to the standards of peer review: panels have neither the breadth of expertise nor the time.

So why is the title of my post “In Defence of REF”? Have I gone “REF Happy” and will soon be under medication, after being found running down the streets of Liverpool naked waving a data stick with over 300 PDFs on it singing “one little star, two little stars,…”.

No.

At least in basic biomedical sciences (a term that covers everything, from biology to maths), I think it is possible to appreciate the quality of a paper in terms of major discovery by reading it, even when the field is not one’s own. Simple questions include “What is the discovery”, “How deep is the discovery”. In a well-written paper, it is obvious after reading the title and the abstract that you should read on to the end: this is a great paper. There will follow papers describing the same phenomenon (in the narrow sense), but in a different system: say a discovery is made in Drosophila, followed by papers in mice and then human cell systems. The first few papers of this sort add to the discovery, so 4* or 3*. When you get the to sixth paper describing the same phenomenon it cannot be 3*, though this may be the one that leads to impact. Research at 2* is important, because it shows the generality of the phenomenon, but it isn’t the discovery. So I would agree that this level of output should have some reward of sorts and that HEFCE has got the reward system wrong. I suspect that the latter reflects pork barrel politics more than anything else.

So the problem may be more for papers that are rather understated or poorly written, where it is only at the end that you realise the importance of the work. Ironically, a considerable number of papers in high profile journals fall into the latter category: figures with multiple panels covering several different experiments, dozens of supplementary figures that are integral to the discovery. They can be very difficult to follow.

THE MOST IMPORTANT benefit of our previous RAE and the REF is the effect on hiring. Decades ago, a significant number of departments would hire the blue eyed boy (rarely a woman) who had just completed a PhD with the Department’s most powerful professor. Sometimes this worked. Often it didn’t and the result was nepotism and a slide into mediocrity, from Nobel Prize to backwater in two generations. I believe that this perpetuated the old boys’ networks and helped to prevent women from entering and rising through the ranks of academia. RAE and REF place a strong selection pressure against hiring in this way and over two decades the changes are clear to see. This is a much, much improved state of affairs. It is closer to a meritocracy than the old system and I believe contributing to the increase in the number of women working their way up through academia.

Metrics for an individual may give an idea of trajectory, but cannot tell us whether a paper is important or not. Only time, which equates to many, many people reading a paper can tell us which papers are important. Thus, the problem remains that there is no way to assess research other than by a panel of people reading to a lesser or greater depth, a pile of papers. I can but hope that panel members are all “Angels”, as defined in Athene Donald’s post on the dramatis personae of panels and read as many of the papers in the pile in front of them as possible whilst remaining clear headed.