Can we evaluate ourselves without journals?

This is a question raised at the end of the excellent article by @Amy_Harmon regarding Open Access and preprints is can biomedical scientists evaluate each other without journals?

The short answer is a resounding yes.  Physical scientists and mathematicians have been posting much of their research as preprints on arXiv for a few decades, with no prejudice to their ability to evaluate the quality of work or of individuals.

The counter argument raised by many in biomedical sciences, from scientists to some journal editors can be boiled down quite simply: We are special and cannot possibly do this.

Various arguments are put forward, from competition (=fear of scooping) to intellectual property. These arguments are heard in many biomedical/biology departments, sometimes leading to quite heated discussions. It is also interesting to note that the defenders of the status quo are not necessarily the older members of the community.

There is a simple answer. Yes you are special, but not in the good sense of the word.

Competition. Here, I fear the answers are rather brutal. If you are in a competitive situation, so a race, then the preprint gives you priority. I would add that most arguments that use competition stem from a lack of creativity. Why on earth are you doing the same work as 6 other labs? The natural world is so vast that a little bit of lateral thinking will rapidly provide many alternatives, which will generate novel insights. There are a very large number of of proteins encoded in the human genome with 0-100 papers. No need to be a sheep. Baaaaaaaaaaa.

IP. This is a non-argument. If there is IP and you wish to protect it, then do so. After this task is completed, you can happily submit preprint and manuscript.

So to return to Amy’s question, an answer by way of anecdote. Dinah Birch, Professor of English at Liverpool University had the unenviable task of running the University’s last REF submission. In my opinion (having had a deep involvement in this) she accomplished this brilliantly and through her leadership delivered a fantastic result (in terms of financial return, which is the name of the game) for the Faculty of Health and Life Sciences (I am not competent to judge how the other Faculties performed). How did she do this? Perhaps not surprisingly given her academic background, she insisted we preformed “Reading Exercises”. Yes, we read our colleagues’ papers, 100s of them.  Our reading provided the necessary information for ranking, which was pretty successful. We even read papers we were unsure about in committee, on a giant screen in the IACD boardroom.

In the end, we read. We cannot use a proxy – the cover or a talk. We have to read. This is what we did as graduate students and what we do as professors. Reading nowadays also means re-analysing data (if these are available). Easy in some fields (e.g., structural biology, genomics), becoming easier in others (e.g., mass spectrometry), almost impossible in many others, but that is changing.

If one tries to use a proxy (journal cover, metrics, name, etc.) as a proxy, one makes horrible mistakes, including in hiring. The clear lesson from data manipulation and ethical scandals, past and brewing, is if you hire on the basis of the cover, your institution is much more likely to have a mess to clear up. My own university had a close call on this front with Melendez – happily he didn’t publish while he was here.

Of course one can make mistakes when reading. However, if reading is ‘public’, that is there is a forum for commenting, such as PubPeer, then while individuals may overlook certain issues and questions, collectively most questions are eventually raised.

So we definitely need reading material to understand a piece of research, but the cover is immaterial.

However, from an aesthetic point of view, the cover, typesetting and so on do add to the reading experience. But here we need to be rational. Few papers are of such scientific and literary merit that they deserve the efforts of skilled typesetters and printers, to then be hard bound in leather before they can grace our desks. A more modest typesetting is pretty cheap and very effective. Have a look at PeerJ who in my experience have the best submission and production set up.