Getting science right side up

There is an argument, which I subscribe to, that the self-righting mechanisms of science are not working. To kick off, it is worth noting that the claim that science is self-righting is made by scientists and by organisations such as journal publishers, whose very existence depends on science. To me there is something of a conflict of interest here.
In my last post on the subject, “Does Science self-right“, I went through a selection of the evidence that highlighted the conflict of interest journals have in claiming that science self-rights and enumerated a number of examples of what we might euphemistically call “problem papers”. Since, I have come across a transcript of Peter Medawar’s excellent broadcast entitled “Is The Scientific Paper a Fraud?“.

This is well worth reading, carefully. It is not about science fraud, but about the false premise upon which we have erected the edifice of the scientific paper as the purveyor of science. His argument is compelling. I would note the start of the second paragraph “Just consider for a moment the traditional form of a scientific paper (incidentally, it is a form which editors themselves often insist upon).”. I would add that we are in danger of slipping to a metaphysical position where we have absolute faith in the paper and believe its contents without question.

Consider.

Retraction Watch has just started a new category, “Doing the right thing“. This is for what we could call “honourable retractions” or “retractions for the right reasons”. Simply put, these are papers that were found to have a flaw (calculation regent, equipment, etc.) and the authors have requested that the paper be retracted. These authors deserve our commiserations, as this is something that can happen to anyone, even the most careful. Accidents will happen.

What is staggering is how few retractions are in this category, just 15. Granted it may be a while before all such retractions are identified in the more than 1200 entries at Retraction Watch, but they are unlikely to make the 5% mark.

So that makes over 1200 retractions for reasons other than genuine error. What sort of reasons? There is a list of categories at Retraction Watch. The category right at the bottom is “Unhelpful Retraction Notices”, which translates roughly as “the Publisher didn’t want to tell its readership why it retracted this paper”. Such a position is in direct contradiction to the notion that science is self-righting, because it prevents debate and challenge.

Another common theme is the use of euphemisms, something I posted about before (Thesaurus of euphemisms). Words such as “misconduct ” and “fraud” are rarely seen in retraction notices (note that Retraction Watch is exactly what it says on the tin – it catalogues retractions, so it is a source of data only). Moreover, abuse of data, either through re-using data to illustrate a different experiment, something I have posted on before (and here and others here), or through heavy cut and paste, as in a recent case at Cancer Cell may well earn a correction rather than a retraction. So there are fewer retractions than warranted. The percentage of retractions due to honest error are in a tiny minority indeed.

The bottom line here is that retractions are under represented in the literature and journals are often coy about the true reason for retraction. This is a long way from self-righting.

So how what are the options before us?

Option 1 Do nothing, acquiesce and allow the corrupt enterprise to prosper. This betrays any notion of integrity and any idea that teaching and mentoring are anything other than cynical grooming activities.

Option 2 Leave. This is the gist of a post by a PhD student at EPFL.

I would note that we do not know whether his thesis was in trouble, but such information has no relevance to the fact that the critiques in the letter stand. The problem is that if all the good go, that leaves only the rotten and science has too large a societal impact for this to be viable. We would end up without science, but living in a world where metaphysics dominates – welcome to the Dark Ages.

Option 3 Do something.
Only this option will get science right side up and ensure that we can proclaim that science is indeed self righting.
However, option 3 has a number of problems. It is time consuming, there is a reluctance of institutions and publishers to respond and, if you are not in a mega secure position, this could potentially cause seriously harm to your research: grant income, bursaries for PhD students and publishing. The solution is anonymity, the subject of a recent article in The Scientist by Kerry Grens on the anonymous whistleblower “Claire Francis”

The importance of anonymity is highlighted by a conversation I had a couple of weeks ago with two young tenured academics, in the course of a Skype discussion over some data analysis. I mentioned that PubPeer is becoming integral to my research groups’s process of trawling the literature. My two colleagues hadn’t heard of PubPeer and were visibly relieved when I told them it was anonymous. Why? Because they could not consider criticising the top of the science pyramid in their country. To do so would result in a swift end to their hard earned research careers.
Does PubPeer work? It seems to. I came across PubPeer not long after it was launched, but didn’t have time to have a good look until very recently. Uptake is growing and there is some excellent discussion, which really illuminates the understanding of the papers.

A new development is that people are starting to take the outputs from their lab journal clubs on PubPeer, here and here.

This is a great idea, you get feedback a lot faster than having to wait until the next conference.

There are some examples of great discussions between “Peers” and authors, e.g., here, here and here. Taking the paper and the discussion together provides a far better understanding of the paper and the considerable strength of the data.

In other instances peers have raised questions about a wide range of issues, including methodology, possible image manipulation (examples here, here, here and here) and re-use of data (here and here), as well as inconsistent data between papers (here). The most discussion I could find was on this paper, which has a companion here.

The next part of this post should, hopefully, go out of date: In none of the above “other instances” I picked up have the authors replied. Indeed, in the paper with the most comments, “Peer 6” is the author of the associated News and Views (not a reviewer of the paper). This has then been followed by a more extensive quote from the News and Views by someone else – in what has been interpreted by some as being suggestive of a damage limitation exercise by NPG in the light of the comments and the publication of clear evidence by a contrary paper by a different group, which is here.

So Option 3 is it, but it is slow, buy in is entirely voluntary and large numbers of authors do not seem to want to participate. They should. If you do not participate, then over time I think this will detract from how the paper is received by the community. This is not a witch hunt, merely the examination of the evidence, the process that underpins science.

To close: PubPeer is currently the only functional service for commenting on a paper. Authors and reader should make full use of it.
When you make comments, regardless of your level of frustration or anger, it is important keep your language correct and precise at all times. After all that is why we are scientists, the world is full of wonders that continue to surprise. I, for one, look forward to growing engagement with PubPeer and so to increased debate and discussion.

Update 3 November 2013. A restricted commenting system is now operating at PubMed. This may, in due course roll to to allow greater access, at least to the comments; commenting may remain restricted due to workload issues at NCBI.