The real problems of post publication peer review

Discussion surrounding post publication peer review (previous post here seems to be growing and one issue that is frequently raised is anonymity. In a PLOS Medicine editorial Hilda Bastian argues that current post publication peer review is over focussed on what apparently is wrong in papers and that anonymity is a threat to effective post publication peer review.
A PubPeer thread takes issue with these and some other points and I have also joined in (I am Peer2). We should remember that any notion of power has nothing to do with scientific capability – indeed there may even be an inverse relation. So providing those with the least power (so the most disenfranchised) a means to participate in post publication peer review is essential. Though we have no data on PubPeer, PubMed Commons is a venue for the established. There are some critiques, there is also a fair amount of hagiography too. I would hazard a guess that PubPeer is far more diverse in terms of the career stage of participants and in terms of their gender/social group. Certainly my anecdotal evidence suggests as much, and that is all I have to go on.

Anonymity is also linked to the idea that post publication peer review is heading into a swamp of negativity. Constructive engagement means that a paper has an impact on your research. That is, the data and/or their interpretation alter your world view. In other words a good paper. What happens next? This may range from seeing your own data differently to collaboration. The latter may involve putting together a new synthesis of a field, something I am attempting to do with colleagues. This paticular new synthesis will be published (open access, naturally) and will hopefully garner comments, critique and so on. We have new platforms for this, e.g., The Winnower, Science Open, which I find very exciting.

There are other forms of constructive engagement, and I would put my comment here and here in this category.

The above illustrates the major problem facing post publication peer review: a lack of engagement by authors. Even an acknowledgement along the lines of “Thanks, we are thinking about this” is unusual.

There is no need to reward the transaction from the point of view of the person posting a comment – critical reading is part of our daily fare. There is no need to reward the authors of the paper: they have a publication. There is a need for authors to realise that they absolutely must engage, otherwise their work will lose credibility. I think the problem of lack of engagement is so substantial that currently author engagement is the surprise, not the norm. Recent examples of author engagement are seen in the threads relating to a paper by Sophien Kamoun (it is worth noting that after what must have been an initial shock Sophien Kamoun and his lab were encouraged by the positive response of peers), and by Jim Carrington.

So critiques are just that. Civility and tone lower a little at times when there is no engagement by authors and peers find that there isn’t just one paper, but a set by an author with analogous critiques. This is certainly common in threads relating to multiple instances of gel splicing (why I think this particular practice is wrong is here). Nonetheless, the frustration of peers is kept pretty much under control, to their credit

I think that this is reasonable, after all we are engaged in science, not in a matter of faith, though as ever, people should stick to the data and let them speak for themselves. There are a number of labs whose papers I not longer bother reading, precisely because a substantial part of their oeuvre has important critiques, mirrored across multiple papers with no response. This looks to me like a potential class of “Pathological Science” and there is so much to read that I need as many filters as possible.

The real problems with anonymity are sockpuppetry and gish galloping, which occur more frequently than incivility. As Peer1 pointed out these are unavoidable. They are easy to identify and so one is free to enjoy the lengths people may go to defend the indefensible (see here for an example) or to skip to the next comment in the thread.

Anonymity is good, needs protecting and under threat. Lets keep it until such time as science no longer needs it.