Reading my colleagues’ papers, a key step in our evaluation of outputs, was very, very useful. I learned many new things and also understand the research across my Institute and indeed large parts of the Faculty much better. The University has yet to use this knowledge accumulated by its REF Wallahs – it would make sense for these people to inform research policy, they have read a lot of papers.

The Faculty Clinical Medicine subpanel meetings were fun – my colleagues on this panel are a great bunch to work with and their injection of humour into proceedings eased the pain.

What about the “pain”? The combination of being REF Wallah and Head of Department came close to killing my research career – 2013 outputs were down to just 2 papers, 2014 is a more reasonable 6, and 2015 will be similar. So REF is destructive at a personal level and consequently at an institutional level.

Is the definitively better research performance of the institute in 2014 compared to the 1990s and 2000s down to RAE/REF ?

I argued in a previous post “In Defence of REF” that RAE/REF have been important in combating nepotism in appointments and ensuring the adoption of a meritocratic agenda in appointments. So one can argue that there is a correlation between the idea of RAE/REF exerting such pressure and those departments that have improved by pursing a meritocratic appointment agenda.

However, this is a correlation

Would that agenda have been pushed anyway, simply due to the normal pressures associated with managing a department in a university? I suspect the answer is at least a partial yes. Looking across the time covered by RAE/REF there are examples, such as ours, of departments that have increased in research power, but also examples where the reverse happened, due in part to poor appointments. So perhaps the value of the pressure exerted by RAE/REF in increasing research activity is overvalued?

I don’t have a clear answer here and I haven’t come across a suitable longitudinal study that might provide data to develop a clear answer. However, from my limited personal view across various departments in the UK where I have some in-depth knowledge, the answer can at most be a qualified yes, and no more.

The other side of the coin is the view that some sort of external pressure is always good, but when it becomes all consuming its value is reduced to zero, and then it becomes a hindrance to progress. RAE/REF looks like it has run its course and may well have entered negative value territory.

However, we need to be answerable to Parliament and hence the taxpayer. So some sort of method is required that provides funding to universities that is not tied to specific PIs, to allow universities to function as such.

There are many alternatives. Given REF 2014 panels that covered large areas of research, then for STEM fields, Dorothy Bishop’s idea of a Departmental h-index seems quite useful – we get a result, it is cheap and staff can get on with research and teaching, rather than sinking without trace in the swamp of REF. Another way forward is to consider what drives research performance down or up. Ultimately it is people and an institution’s ability to foster long-term creativity in staff.

So down drivers include: insularity, nepotism in appointments, discrimination, bullying, and so on.

Up drivers: expansiveness, meritocratic hiring with a 30+ year perspective (=career), diversity and mentoring.

A system proposed by Jenny Martin, provides an alternative set of parameters and these certainly fit the bill for driving research performance up.

So have RAE/REF been useful? On balance, I think not. There are easier ways to distribute funds from government to universities and I cannot see that the assessments have in themselves directly driven an increase in research quality.

David Paramelle’s paper on using gold nanoparticles stoichiometrically functionalised with a peptide that recognises sphingolipids has just been published in Advanced Healthcare Materials (Publisher’s site; Pubmed)

The paper is the classic “Sunday afternoon” project, which arose through discussions with Rachel Kraut at NTU.

As ever, a lot more than Sunday afternoons ended up being put into the paper, because David had to develop some new approaches. Particularly nice was the purification of nanoparticles functionalised with the sphingomyelin-binding peptide (called “SBD”) from non-functionalised nanoparticles. This is a key step for the preparation of nanoparticles carrying just one functional peptide or group. Hitherto, we have happily had affinity tags as the functional group, which allows for affinity chromatography (examples here, here and here).

With the SBD peptide there was no such opportunity, so David figured a way using ion-exchange. Specific purification by ion-exchange of nanoparticles carrying a charged function has often been difficult to implement on nanoparticles, perhaps because of interactions between the functional part of the ligand and the underlying ligand shell. However, in this case it worked very nicely. Result, nanoparticles functionalised with a single SBD peptide, able to recognise sphingolipid enriched membrane domains (lipid rafts), at a sensitivity far higher than fluorescently-labelled SBD peptide.

Quentin Nunes, PhD

Congratulations to Quentin Nunes, who today successfully defended his PhD today. His first paper from his thesis work was published in late 2013 in Pancreatology. This was an analysis, using public datasets of mRNA expression data, of the putative heparin-binding protein network in the healthy pancreas and in pancreatic digestive diseases. The latter part of his thesis work will be submitted for publication soon (watch this space!) and is a proteomics analysis of heparin-binding proteins in mouse pancreas and in a mouse model of acute pancreatitis.

Discussion surrounding post publication peer review (previous post here seems to be growing and one issue that is frequently raised is anonymity. In a PLOS Medicine editorial Hilda Bastian argues that current post publication peer review is over focussed on what apparently is wrong in papers and that anonymity is a threat to effective post publication peer review.
A PubPeer thread takes issue with these and some other points and I have also joined in (I am Peer2). We should remember that any notion of power has nothing to do with scientific capability – indeed there may even be an inverse relation. So providing those with the least power (so the most disenfranchised) a means to participate in post publication peer review is essential. Though we have no data on PubPeer, PubMed Commons is a venue for the established. There are some critiques, there is also a fair amount of hagiography too. I would hazard a guess that PubPeer is far more diverse in terms of the career stage of participants and in terms of their gender/social group. Certainly my anecdotal evidence suggests as much, and that is all I have to go on.

Anonymity is also linked to the idea that post publication peer review is heading into a swamp of negativity. Constructive engagement means that a paper has an impact on your research. That is, the data and/or their interpretation alter your world view. In other words a good paper. What happens next? This may range from seeing your own data differently to collaboration. The latter may involve putting together a new synthesis of a field, something I am attempting to do with colleagues. This paticular new synthesis will be published (open access, naturally) and will hopefully garner comments, critique and so on. We have new platforms for this, e.g., The Winnower, Science Open, which I find very exciting.

There are other forms of constructive engagement, and I would put my comment here and here in this category.

The above illustrates the major problem facing post publication peer review: a lack of engagement by authors. Even an acknowledgement along the lines of “Thanks, we are thinking about this” is unusual.

There is no need to reward the transaction from the point of view of the person posting a comment – critical reading is part of our daily fare. There is no need to reward the authors of the paper: they have a publication. There is a need for authors to realise that they absolutely must engage, otherwise their work will lose credibility. I think the problem of lack of engagement is so substantial that currently author engagement is the surprise, not the norm. Recent examples of author engagement are seen in the threads relating to a paper by Sophien Kamoun (it is worth noting that after what must have been an initial shock Sophien Kamoun and his lab were encouraged by the positive response of peers), and by Jim Carrington.

So critiques are just that. Civility and tone lower a little at times when there is no engagement by authors and peers find that there isn’t just one paper, but a set by an author with analogous critiques. This is certainly common in threads relating to multiple instances of gel splicing (why I think this particular practice is wrong is here). Nonetheless, the frustration of peers is kept pretty much under control, to their credit

I think that this is reasonable, after all we are engaged in science, not in a matter of faith, though as ever, people should stick to the data and let them speak for themselves. There are a number of labs whose papers I not longer bother reading, precisely because a substantial part of their oeuvre has important critiques, mirrored across multiple papers with no response. This looks to me like a potential class of “Pathological Science” and there is so much to read that I need as many filters as possible.

The real problems with anonymity are sockpuppetry and gish galloping, which occur more frequently than incivility. As Peer1 pointed out these are unavoidable. They are easy to identify and so one is free to enjoy the lengths people may go to defend the indefensible (see here for an example) or to skip to the next comment in the thread.

Anonymity is good, needs protecting and under threat. Lets keep it until such time as science no longer needs it.

A milestone

Sometime last night this blog received its 50,000th page view. I write this blog because I like to. That others find the content worth reading at times is lovely, thank you.

What has been read the most and the least? Continue Reading »

Leonid Schneider has a guest post on Retraction Watch “What if universities had to agree to refund grants whenever there was a retraction?” that has generated a lot of discussion. My own comment became so long that I am posting it below. For those who are not aware, in the USA, the Office of Research Integrity (ORI) has the power to reclaim from institutions grant funding acquired through fraudulent means, e.g., manipulated or made up data, though there is a time limit and this is only exerted in a fraction of the cases investigated by ORI. No other country has a similar or analogous mechanism.

I like Leonid Schneider’s idea. Continue Reading »

REF 2014 is the fourth assessment in a row where we have made real progress, and this is very pleasing.
In RAE 1996 the newly formed School of Biological Sciences attained a grade 4, which equates to research of national standard, with some research of international standard.
In RAE 2001 44/65 staff were returned, attaining a grade 5B. This equates to over 50% (and we were only just over the 50% bar on this one) of returned staff producing research of international standard. In RAE 2008 terms (and in REF 2013) this (more or less) corresponds to 2* to 4*. Though difficult to compare between different grading systems, a guesstimate of the School of Biological Sciences RAE 2001 performance in RAE 2008 terms is ~ 35-40% 2*-4* research, with the balance of 65-60% at 1*.
In RAE 2008 all 65 staff were returned, with 40% 4* and 3*, 45% 2*, 15% 1*, and, as I mentioned in the previous post in REF 2014 we are at least up to 65% of all staff having 3* and 4* outputs.
So progression in score. This also reflects real change, it isn’t cosmetic. It would be obvious to anyone visiting the School/Institute over the 20 years 1994-2014 that the place has changed and there is a far greater buzz about.
This begs a question, which I will turn to in the future post: Was it RAE/REF that drove these changes or something else, and so is RAE/REF worthwhile?


Get every new post delivered to your Inbox.

Join 358 other followers