There is an argument, which I subscribe to, that the self-righting mechanisms of science are not working. To kick off, it is worth noting that the claim that science is self-righting is made by scientists and by organisations such as journal publishers, whose very existence depends on science. To me there is something of a conflict of interest here.
In my last post on the subject, “Does Science self-right“, I went through a selection of the evidence that highlighted the conflict of interest journals have in claiming that science self-rights and enumerated a number of examples of what we might euphemistically call “problem papers”. Since, I have come across a transcript of Peter Medawar’s excellent broadcast entitled “Is The Scientific Paper a Fraud?“.
This is well worth reading, carefully. It is not about science fraud, but about the false premise upon which we have erected the edifice of the scientific paper as the purveyor of science. His argument is compelling. I would note the start of the second paragraph “Just consider for a moment the traditional form of a scientific paper (incidentally, it is a form which editors themselves often insist upon).”. I would add that we are in danger of slipping to a metaphysical position where we have absolute faith in the paper and believe its contents without question.
Consider.
Retraction Watch has just started a new category, “Doing the right thing“. This is for what we could call “honourable retractions” or “retractions for the right reasons”. Simply put, these are papers that were found to have a flaw (calculation regent, equipment, etc.) and the authors have requested that the paper be retracted. These authors deserve our commiserations, as this is something that can happen to anyone, even the most careful. Accidents will happen.
What is staggering is how few retractions are in this category, just 15. Granted it may be a while before all such retractions are identified in the more than 1200 entries at Retraction Watch, but they are unlikely to make the 5% mark.
So that makes over 1200 retractions for reasons other than genuine error. What sort of reasons? There is a list of categories at Retraction Watch. The category right at the bottom is “Unhelpful Retraction Notices”, which translates roughly as “the Publisher didn’t want to tell its readership why it retracted this paper”. Such a position is in direct contradiction to the notion that science is self-righting, because it prevents debate and challenge.
Another common theme is the use of euphemisms, something I posted about before (Thesaurus of euphemisms). Words such as “misconduct ” and “fraud” are rarely seen in retraction notices (note that Retraction Watch is exactly what it says on the tin – it catalogues retractions, so it is a source of data only). Moreover, abuse of data, either through re-using data to illustrate a different experiment, something I have posted on before (and here and others here), or through heavy cut and paste, as in a recent case at Cancer Cell may well earn a correction rather than a retraction. So there are fewer retractions than warranted. The percentage of retractions due to honest error are in a tiny minority indeed.
The bottom line here is that retractions are under represented in the literature and journals are often coy about the true reason for retraction. This is a long way from self-righting.
So how what are the options before us?
Option 1 Do nothing, acquiesce and allow the corrupt enterprise to prosper. This betrays any notion of integrity and any idea that teaching and mentoring are anything other than cynical grooming activities.
Option 2 Leave. This is the gist of a post by a PhD student at EPFL.
I would note that we do not know whether his thesis was in trouble, but such information has no relevance to the fact that the critiques in the letter stand. The problem is that if all the good go, that leaves only the rotten and science has too large a societal impact for this to be viable. We would end up without science, but living in a world where metaphysics dominates – welcome to the Dark Ages.
Option 3 Do something.
Only this option will get science right side up and ensure that we can proclaim that science is indeed self righting.
However, option 3 has a number of problems. It is time consuming, there is a reluctance of institutions and publishers to respond and, if you are not in a mega secure position, this could potentially cause seriously harm to your research: grant income, bursaries for PhD students and publishing. The solution is anonymity, the subject of a recent article in The Scientist by Kerry Grens on the anonymous whistleblower “Claire Francis”
The importance of anonymity is highlighted by a conversation I had a couple of weeks ago with two young tenured academics, in the course of a Skype discussion over some data analysis. I mentioned that PubPeer is becoming integral to my research groups’s process of trawling the literature. My two colleagues hadn’t heard of PubPeer and were visibly relieved when I told them it was anonymous. Why? Because they could not consider criticising the top of the science pyramid in their country. To do so would result in a swift end to their hard earned research careers.
Does PubPeer work? It seems to. I came across PubPeer not long after it was launched, but didn’t have time to have a good look until very recently. Uptake is growing and there is some excellent discussion, which really illuminates the understanding of the papers.
A new development is that people are starting to take the outputs from their lab journal clubs on PubPeer, here and here.
This is a great idea, you get feedback a lot faster than having to wait until the next conference.
There are some examples of great discussions between “Peers” and authors, e.g., here, here and here. Taking the paper and the discussion together provides a far better understanding of the paper and the considerable strength of the data.
In other instances peers have raised questions about a wide range of issues, including methodology, possible image manipulation (examples here, here, here and here) and re-use of data (here and here), as well as inconsistent data between papers (here). The most discussion I could find was on this paper, which has a companion here.
The next part of this post should, hopefully, go out of date: In none of the above “other instances” I picked up have the authors replied. Indeed, in the paper with the most comments, “Peer 6” is the author of the associated News and Views (not a reviewer of the paper). This has then been followed by a more extensive quote from the News and Views by someone else – in what has been interpreted by some as being suggestive of a damage limitation exercise by NPG in the light of the comments and the publication of clear evidence by a contrary paper by a different group, which is here.
So Option 3 is it, but it is slow, buy in is entirely voluntary and large numbers of authors do not seem to want to participate. They should. If you do not participate, then over time I think this will detract from how the paper is received by the community. This is not a witch hunt, merely the examination of the evidence, the process that underpins science.
To close: PubPeer is currently the only functional service for commenting on a paper. Authors and reader should make full use of it.
When you make comments, regardless of your level of frustration or anger, it is important keep your language correct and precise at all times. After all that is why we are scientists, the world is full of wonders that continue to surprise. I, for one, look forward to growing engagement with PubPeer and so to increased debate and discussion.
Update 3 November 2013. A restricted commenting system is now operating at PubMed. This may, in due course roll to to allow greater access, at least to the comments; commenting may remain restricted due to workload issues at NCBI.
Amen to that. The system is utterly, completely broken. Personally, I lay part of the blame firmly at the feet of journal editors, who simply refuse to “grow a pair” and admit there’s a problem. There have been some retractions in high profile journals I’ve been involved with recently, in which there’s clearly a back story involving less than honest scientific practices, but for various reasons (usually under threat of lawsuits from the authors) the journal simply allows the authors to get away with a “this article has been withdrawn” statement. Similarly there are other cases where journals go along with the findings of institutional investigations, which are (to put it politely) conflicted out the wazoo. In other cases, there have been corrections that were allowed in between the in-press and the in-print versions, but never documented on the journal site or in PubMed. Yet more cases in which journals claim to have looked into something and found no fault, but the clearly-faked data is still out there for all to see. Then there’s the whole anonymity issue (see article in The Scientist this week), the conclusion of which is essentially “we’ll deal with allegations if we feel like it, but reserve the right not to if you’re on our blacklist (Clare Francis)”. Don’t get me started on the whole issue of how long these things take – in my experience it’s usually 20+ emails and about 6 months to get any action, and 6 months is fast these days (I know of ongoing cases which have dragged out for 2-3 years so far, “awaiting results of an independent investigation”. Then of course there’s the problem of PubMed being short-staffed and delays in making sure everything is visible and open and linked between journal websites and PubMed. Then there’s COPE, which has no teeth… so many journals who claim to be members of COPE violate their guidelines on a regular basis, it’s laughable. And of course, the upstart PubPeer, which is doing great things, but is a self-funded group of young PI’s who are too scared to reveal who they are. You really have to ask why the journals aren’t clamoring for a piece of the action? Why aren’t C/N/S and the other glamor mags offering to fully fund PubPeer and take it over? Perhaps because the entire thing makes a mockery of the veneer of reputation they’ve built over the years. The first step is for COPE/PubPeer and the journals to sit around a table and admit there’s a problem, and come up with a solution with teeth. One potential solution – if your journal has a poor “pubpeer ranking” for responsiveness to allegations of misconduct, maybe publications in your journal should not be submissible for credit on NIH progress reports. Watch how quickly NIH grantees start sending their work to places that have better accountability scores, once they learn their renewal funding will be penalized. It would take 5 years to take effect, but that’s not long to build a better system.
Tl/DR there’s the problem of misconduct per se, but mix it together with a hopelessly fragmented and inadequate system for dealing with misconduct, and the results are pretty shocking.
Update from Ivan Oransky regarding the figures at Retraction Watch. The figures I quote for the total number of retractions are a little off. In some instances a single retraction has several posts, though in perhaps fewer cases, a single post may have more than one retraction. In addition, Retraction Watch has some posts dealing with news on retractions. The number will still be well above the 1000 mark though. If I may allow myself a wry smile and be so bold as to coin a phrase, “this additional precision does not change the conclusions”.
[…] « Getting science right side up […]
[…] education, peer review is a continuous process. PubPeer, which I have mentioned before (here and here) provides a route for post publication peer review. Activity at PubPeer is increasing, which is […]
[…] science remains “disorganised” and, most crucially self-righting (posts on the latter here and […]
[…] while back I did a rough calculation based on the total number of retractions at RW and the fraction that were […]