• Home
  • Development of Heparin and HS Glycotherapeutics
  • Lab
  • People

Ferniglab Blog

The personal blog of Dave Fernig, thoughts on science and unrelated matters

Feeds:
Posts
Comments
« One future of science publishing
“Analysis of the FGFR Signalling Network with Heparin as Co-Receptor” »

In Defence of REF

April 20, 2013 by ferniglab


I never thought I would write a post with this title and, after last week, it is even more surprising. Nonetheless, I am a modest fan of REF and this post sets out why.

First, what might be construed as my (many) conflicts of interest.
I am partly responsible for our Faculty’s submission to one of the Research Excellence Framework (REF) panels. Last week science was my displacement activity, as I collated and moderated the results of our last REF reading exercise. Eight papers chosen by each member of staff in consultation with their head of Department, each read by two professors (I was one of these too). Where grading or its justification was contradictory, a third reader stepped into the breach. After moderating the choice of the best four out of the eight, I moderated the likelihood of a member of staff being returned or not. I estimate a good 30 h just for the moderation and collation. Maybe I take it too seriously, but in essence that is what it means (or should do in my mind) to be a professor. Your research at times seems like a bit of a hobby, as you help deliver the goods for the institution.

The concerns regarding how panels will deal with the mass of papers have been voiced repeatedly. Most recently in Philip Moriarty’s post at IOP
and at Telescoper’s blog. I agree with both analyses.

We are reassured by HEFCE, RCUK and panel chairs that citation data will either not be used or be used with a light touch. Light touch is presumably to push a paper up, not down? So they will be reading the papers. Can they do this? Telescoper’s post clearly demonstrates that panel members cannot read the papers according to the standards of peer review: panels have neither the breadth of expertise nor the time.

So why is the title of my post “In Defence of REF”? Have I gone “REF Happy” and will soon be under medication, after being found running down the streets of Liverpool naked waving a data stick with over 300 PDFs on it singing “one little star, two little stars,…”.

No.

At least in basic biomedical sciences (a term that covers everything, from biology to maths), I think it is possible to appreciate the quality of a paper in terms of major discovery by reading it, even when the field is not one’s own. Simple questions include “What is the discovery”, “How deep is the discovery”. In a well-written paper, it is obvious after reading the title and the abstract that you should read on to the end: this is a great paper. There will follow papers describing the same phenomenon (in the narrow sense), but in a different system: say a discovery is made in Drosophila, followed by papers in mice and then human cell systems. The first few papers of this sort add to the discovery, so 4* or 3*. When you get the to sixth paper describing the same phenomenon it cannot be 3*, though this may be the one that leads to impact. Research at 2* is important, because it shows the generality of the phenomenon, but it isn’t the discovery. So I would agree that this level of output should have some reward of sorts and that HEFCE has got the reward system wrong. I suspect that the latter reflects pork barrel politics more than anything else.

So the problem may be more for papers that are rather understated or poorly written, where it is only at the end that you realise the importance of the work. Ironically, a considerable number of papers in high profile journals fall into the latter category: figures with multiple panels covering several different experiments, dozens of supplementary figures that are integral to the discovery. They can be very difficult to follow.

THE MOST IMPORTANT benefit of our previous RAE and the REF is the effect on hiring. Decades ago, a significant number of departments would hire the blue eyed boy (rarely a woman) who had just completed a PhD with the Department’s most powerful professor. Sometimes this worked. Often it didn’t and the result was nepotism and a slide into mediocrity, from Nobel Prize to backwater in two generations. I believe that this perpetuated the old boys’ networks and helped to prevent women from entering and rising through the ranks of academia. RAE and REF place a strong selection pressure against hiring in this way and over two decades the changes are clear to see. This is a much, much improved state of affairs. It is closer to a meritocracy than the old system and I believe contributing to the increase in the number of women working their way up through academia.

Metrics for an individual may give an idea of trajectory, but cannot tell us whether a paper is important or not. Only time, which equates to many, many people reading a paper can tell us which papers are important. Thus, the problem remains that there is no way to assess research other than by a panel of people reading to a lesser or greater depth, a pile of papers. I can but hope that panel members are all “Angels”, as defined in Athene Donald’s post on the dramatis personae of panels and read as many of the papers in the pile in front of them as possible whilst remaining clear headed.

Advertisement

Share this:

  • Twitter
  • Facebook
  • LinkedIn

Like this:

Like Loading...

Related

Posted in Science process, Scientific progress | Tagged science, Science progress | 3 Comments

3 Responses

  1. on April 21, 2013 at 11:00 am REF again! | To the left of centre

    […] the forthcoming Research Excellence Framework (REF2014). An interesting post by Dave Fernig called In Defence of REF. This post does make some valid points. REF, and previous RAEs, may well have encouraged more […]


  2. on August 7, 2013 at 11:01 am REF: !#?! GRRRR, Aaaaaarrrrrrrgh | Ferniglab's Blog

    […] late April, I posted “In Defence of REF” to highlight the positives of the UK’s assessment of research, though I did temper […]


  3. on May 7, 2015 at 3:35 pm REF 2014 and all that (III): is RAE/REF a driver or passenger of change? | Ferniglab Blog

    […] argued in a previous post “In Defence of REF” that RAE/REF have been important in combating nepotism in appointments and ensuring the adoption […]



Comments are closed.

  • Places of interest

    The one and only PhD comics, the guide to being a graduate and to mentoring.

    Improbable Research and the Ig Nobels

    Retraction Watch provides updates on retractions of articles.

    Office for Research Integrity, their video should be compulsory for all.

    Centre for Alternative Technology

    Lateral Science, has some quite stunning information - well worth a browse.

    Fascinating places that have been closed by lawyers

    Science Fraud, shut down due to legal threats on Jan 3 2013. and Abnormal Science

  • Blogroll

    • WordPress.com
    • WordPress.org
  • Funding agencies

    • Biotechnology and Biological Sciences Research Council
    • Cancer and Polio Research Fund
    • Engineering and Physical Sciences Research Council
    • Liverpool Pancreas NIHR Biomedical Research Unit
    • Medical Research Council
    • North West Cancer Research
  • Seminars

    • Cancer Research UK Centre
  • April 2013
    M T W T F S S
    1234567
    891011121314
    15161718192021
    22232425262728
    2930  
    « Mar   May »
  • Archives

    • November 2022
    • July 2022
    • April 2022
    • March 2022
    • May 2021
    • March 2021
    • August 2020
    • June 2020
    • April 2020
    • March 2020
    • December 2019
    • October 2019
    • July 2019
    • May 2019
    • April 2019
    • January 2019
    • September 2018
    • August 2018
    • May 2018
    • April 2018
    • February 2018
    • January 2018
    • December 2017
    • November 2017
    • July 2017
    • June 2017
    • April 2017
    • March 2017
    • January 2017
    • October 2016
    • June 2016
    • May 2016
    • March 2016
    • January 2016
    • December 2015
    • July 2015
    • June 2015
    • May 2015
    • March 2015
    • February 2015
    • January 2015
    • December 2014
    • November 2014
    • October 2014
    • September 2014
    • August 2014
    • July 2014
    • June 2014
    • February 2014
    • January 2014
    • December 2013
    • October 2013
    • September 2013
    • August 2013
    • July 2013
    • June 2013
    • May 2013
    • April 2013
    • March 2013
    • February 2013
    • January 2013
    • December 2012
    • November 2012
    • October 2012
    • September 2012
    • August 2012
    • July 2012
    • June 2012
    • May 2012
    • April 2012
    • March 2012
    • February 2012
    • January 2012
    • November 2011
    • October 2011
    • September 2011
    • August 2011
    • July 2011
    • June 2011
    • May 2011
    • April 2011
    • March 2011
    • February 2011
    • January 2011
    • December 2010
    • November 2010
    • October 2010
    • September 2010
    • August 2010
  • Follow me on Twitter

    My Tweets
  • Cloud

    American Civil War antithrombin III banana Biological imaging biotechnology Brexit Choanoflagellates chocolate chondroitin sulfate coagulation Confederate States covid19 DN Lee Education EU EU referendum Europe extracellular matrix FGF Fibroblast growth factor Food FRET sensors Gish Gallop glycosaminoglycans GMO government Graduate students heparan sulfate heparin history of science imaging Irvine Stephens Bulloch James Bulloch James Dunwoody Bulloch Liverpool microbiology Nanoparticle Nanoparticles Nanotechnology neuroscience nmr Open Access Open Data orange Parliament Peer Review PhD polysaccharide port sunlight Post publication peer review protein chemistry REF research Research Excellence Framework Research integrity Roast SARS-CoV-2 science Science and Technology Committee Science fraud Science Funding Science progress Scientific American Seminars sorbet speaking strawberry sulfation Sulfotransferase synthetic biology Teaching technology transfer Tourism Travel Universities

Blog at WordPress.com.

WPThemes.


Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy
  • Follow Following
    • Ferniglab Blog
    • Join 73 other followers
    • Already have a WordPress.com account? Log in now.
    • Ferniglab Blog
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Copy shortlink
    • Report this content
    • View post in Reader
    • Manage subscriptions
    • Collapse this bar
%d bloggers like this: