Congratulations to Quentin Nunes, who today successfully defended his PhD today. His first paper from his thesis work was published in late 2013 in Pancreatology. This was an analysis, using public datasets of mRNA expression data, of the putative heparin-binding protein network in the healthy pancreas and in pancreatic digestive diseases. The latter part of his thesis work will be submitted for publication soon (watch this space!) and is a proteomics analysis of heparin-binding proteins in mouse pancreas and in a mouse model of acute pancreatitis.
Discussion surrounding post publication peer review (previous post here seems to be growing and one issue that is frequently raised is anonymity. In a PLOS Medicine editorial Hilda Bastian argues that current post publication peer review is over focussed on what apparently is wrong in papers and that anonymity is a threat to effective post publication peer review.
A PubPeer thread takes issue with these and some other points and I have also joined in (I am Peer2). We should remember that any notion of power has nothing to do with scientific capability – indeed there may even be an inverse relation. So providing those with the least power (so the most disenfranchised) a means to participate in post publication peer review is essential. Though we have no data on PubPeer, PubMed Commons is a venue for the established. There are some critiques, there is also a fair amount of hagiography too. I would hazard a guess that PubPeer is far more diverse in terms of the career stage of participants and in terms of their gender/social group. Certainly my anecdotal evidence suggests as much, and that is all I have to go on.
Anonymity is also linked to the idea that post publication peer review is heading into a swamp of negativity. Constructive engagement means that a paper has an impact on your research. That is, the data and/or their interpretation alter your world view. In other words a good paper. What happens next? This may range from seeing your own data differently to collaboration. The latter may involve putting together a new synthesis of a field, something I am attempting to do with colleagues. This paticular new synthesis will be published (open access, naturally) and will hopefully garner comments, critique and so on. We have new platforms for this, e.g., The Winnower, Science Open, which I find very exciting.
The above illustrates the major problem facing post publication peer review: a lack of engagement by authors. Even an acknowledgement along the lines of “Thanks, we are thinking about this” is unusual.
There is no need to reward the transaction from the point of view of the person posting a comment – critical reading is part of our daily fare. There is no need to reward the authors of the paper: they have a publication. There is a need for authors to realise that they absolutely must engage, otherwise their work will lose credibility. I think the problem of lack of engagement is so substantial that currently author engagement is the surprise, not the norm. Recent examples of author engagement are seen in the threads relating to a paper by Sophien Kamoun (it is worth noting that after what must have been an initial shock Sophien Kamoun and his lab were encouraged by the positive response of peers), and by Jim Carrington.
So critiques are just that. Civility and tone lower a little at times when there is no engagement by authors and peers find that there isn’t just one paper, but a set by an author with analogous critiques. This is certainly common in threads relating to multiple instances of gel splicing (why I think this particular practice is wrong is here). Nonetheless, the frustration of peers is kept pretty much under control, to their credit
I think that this is reasonable, after all we are engaged in science, not in a matter of faith, though as ever, people should stick to the data and let them speak for themselves. There are a number of labs whose papers I not longer bother reading, precisely because a substantial part of their oeuvre has important critiques, mirrored across multiple papers with no response. This looks to me like a potential class of “Pathological Science” and there is so much to read that I need as many filters as possible.
The real problems with anonymity are sockpuppetry and gish galloping, which occur more frequently than incivility. As Peer1 pointed out these are unavoidable. They are easy to identify and so one is free to enjoy the lengths people may go to defend the indefensible (see here for an example) or to skip to the next comment in the thread.
Anonymity is good, needs protecting and under threat. Lets keep it until such time as science no longer needs it.
Sometime last night this blog received its 50,000th page view. I write this blog because I like to. That others find the content worth reading at times is lovely, thank you.
What has been read the most and the least? Continue Reading »
Posted in Blogosphere, Post publication peer review, Science process, Science publishing, Scientific progress | Tagged FGF, Fibroblast growth factor, Nanoparticle, Nanoparticles, research, Research integrity, Science fraud, Science progress | Leave a Comment »
Leonid Schneider has a guest post on Retraction Watch “What if universities had to agree to refund grants whenever there was a retraction?” that has generated a lot of discussion. My own comment became so long that I am posting it below. For those who are not aware, in the USA, the Office of Research Integrity (ORI) has the power to reclaim from institutions grant funding acquired through fraudulent means, e.g., manipulated or made up data, though there is a time limit and this is only exerted in a fraction of the cases investigated by ORI. No other country has a similar or analogous mechanism.
I like Leonid Schneider’s idea. Continue Reading »
Posted in Post publication peer review, Research integrity, Science process, Science publishing, Scientific progress | Tagged research, Research integrity, science, Science fraud, Science progress | Leave a Comment »
REF 2014 is the fourth assessment in a row where we have made real progress, and this is very pleasing.
In RAE 1996 the newly formed School of Biological Sciences attained a grade 4, which equates to research of national standard, with some research of international standard.
In RAE 2001 44/65 staff were returned, attaining a grade 5B. This equates to over 50% (and we were only just over the 50% bar on this one) of returned staff producing research of international standard. In RAE 2008 terms (and in REF 2013) this (more or less) corresponds to 2* to 4*. Though difficult to compare between different grading systems, a guesstimate of the School of Biological Sciences RAE 2001 performance in RAE 2008 terms is ~ 35-40% 2*-4* research, with the balance of 65-60% at 1*.
In RAE 2008 all 65 staff were returned, with 40% 4* and 3*, 45% 2*, 15% 1*, and, as I mentioned in the previous post in REF 2014 we are at least up to 65% of all staff having 3* and 4* outputs.
So progression in score. This also reflects real change, it isn’t cosmetic. It would be obvious to anyone visiting the School/Institute over the 20 years 1994-2014 that the place has changed and there is a far greater buzz about.
This begs a question, which I will turn to in the future post: Was it RAE/REF that drove these changes or something else, and so is RAE/REF worthwhile?
An excellent article on the coming of age of post publication peer review by Emma Stoye is up at Chemistry World
She quotes me (correctly) as stating that “Science does not exist without post publication peer review. If anyone wants to follow the quote up, my own posts can be found here.
Now that the dust has settled, institutions have posted their interpretations of the results of the 2014 Research Excellence Framework (REF), and people have gone through the results and institutional interpretations in various blogs, I thought it time to put my oar in. This is the first of a series of posts that will look at the REF 2014 score of the Institute of Integrative Biology, how RAE/REF scores for this academic enterprise have fared over time, and whether REF/RAE is of any use. Continue Reading »