Opening up peer review: the peculiar case of PNAS contributed papers

Proceedings of the National Academy of Sciences (PNAS) has two paths for submission of research articles, one standard and one less so, the famous contributed track where the submitting author has to be a member of the National Academy of Sciences of the United States of America. Peter Aldhous reviewed in 2014 this inside track and those who use it more often. He describes the contributed track as follows: “This unusual process allows authors to choose who will review their paper and how to respond to those reviewers’ comments“.

There are two elements of transparency and accountability to counterbalance this conflict of interest of having an author acting as the editor of their own article: it is recorded on the paper that it is a contributed paper, and, the names of the referees (chosen by the author) are also published. It is interesting (maybe) to note that this ancient house of PNAS has a system there which is pretty similar to what has been recently proposed as a disruptive innovation in scientific publishing by Jan Velterop and implemented by ScienceOpen, i.e. peer review by endorsement (PRO). There are a couple of differences. The first one is that PRO at ScienceOpen is open to everyone, not just National Academy members. The second one is that not just the names of the reviewers, but also the content of the reviews that is shared in PRO.

As readers of this blog will know, David Mason and I have recently challenged a particular contributed PNAS paper by the Mirkin group on StickyFlares. The discussion can be found at PubPeer (the authors did not engage). We requested some data which (after some efforts) we eventually obtained. We wrote a letter to the Editor. Which was eventually rejected by Inder Verma editor of PNAS. The letter is available at BiorXiv.

Given that neither PubPeer nor the PNAS letter to editor enabled to get any answers from the authors to our substantial criticism, we were curious to know if any of the referees had maybe raised similar issues and, if yes, how the authors had replied. Dave therefore wrote to the referees to ask whether they would share their reports. The response was negative; they could not share their reports because “referee comments from peer review have to be kept confidential as it is an essential part of maintaining the integrity of the peer review process“. I was rather surprised by this response and was moved to write the following:

01/10/2015

Dear Shana, Chris

I am Raphael; Dave is a member of my group. Apologies for pitching in and for a rather long response!

Thank you both for your replies. [paragraph edited out about the issue of whether the PNAS guidelines on choice of reviewers were followed; see the PubPeer discussion for more]

I fully share your commitment to defend the integrity of the review process, but I would urge to you to reconsider your decision to keep your comments confidential, precisely because it does not serve that very commendable aim.

It is worth considering for a moment what is the role of confidentiality and anonymity in the peer review process, and also who is in charge of guaranteeing that confidentiality and anonymity. In a traditional peer review process, the reports are confidential and anonymous: the justification is the protection of the reviewers from potential reprisals if they were to write a very critical review. The editors are in charge of protecting the confidentiality and anonymity: this is part of the contract between the editor and the reviewers. If the editor was to publish the reviews or/and, worse, reveal their identity, he/she would breach that contract and this would significantly affect the trust between future reviewers and the journal. I have myself pondered on publishing on my blog the reviews of a (rejected) submission of one of my papers, and was eventually convinced (though I am still entirely sure this was the right decision) not to do so by the detailed comments of an editor who did point out that the reviewers expected their reviews to remain confidential and that I would therefore breach their trust by doing so [1]. It is however a completely different matter for reviewers who can decide to forego their right to anonymity both immediately at the stage of the review process (“signing reviews”, usually, precisely with the motivation of increasing transparency and integrity of the review process), or later, for various reasons (nearly 200 000 examples at Publons, a site that enables and encourages reviewers to share their reports [2-3]). A recent prominent example of a reviewer sharing her reviews (on PubPeer) is Vicki Vance, who had reviewed several of the papers of Olivier Voinnet and noted serious problems (they were nevertheless published) [4]. I have never heard anyone suggesting that a reviewer who would decide to share their reviews of a paper after publication, i.e. their own scholarly evaluation of published work, would be damaging the integrity of the peer review process. I also really fail to see by what mechanism it could do so.

Obviously, the PNAS “contributed submission” path is another can of worms. Many would argue that it is in itself damaging to the integrity of the peer review process with this very unusual situation where an author chose its reviewers. In this specific case, it is hard to see any justification at all for the confidentiality of the reviews: it does not serve to protect the reviewers from potential reprisals from the author since the author has chosen its referees in the first place. The only thing it does is prevent the public (and in particular other scientists) to benefit from the insights that would be provided by sharing the reviews. I would argue that here even more than in any other case, sharing the reviews would be the best way to protect the integrity of the peer review process and therefore I hope you will reconsider,

Best wishes

Raphael

[1] https://raphazlab.wordpress.com/2013/01/03/nature-materials-peer-view-of-stripy-revisited-july-to-september-2009-confidential-or-not/ (see first comment in particular)
[2] https://publons.com/
[3] The scientists who get credit for peer review, Nature, 2014
[4] http://www.lab-times.org/editorial/e_600.lasso

Unfortunately, I did not get a reply.

Advertisements

One comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s