I reproduce below a comment I have left on this Nature editorial entitled “Go forth and replicate!“.
Nature Publishing Group encouragement of replications and discussions of their own published studies is a very welcome move. Seven years ago, I wrote a letter (accompanying a submission) to the Editor of Nature Materials. The last paragraph of that letter read: “The possibility of refuting existing data and theories is an important condition of progress of scientific knowledge. The high-impact publication of wrong results can have a real impact on research activities and funding priorities. There is no doubt that the series of papers revisited in this Report contribute to shape the current scientific landscape in this area of science and that their refutation will have a large impact.” [1]
The submission was “Stripy Nanoparticles Revisited” and it took three more years to publish it… in another journal; meanwhile Nature Materials continued to publish findings based on the original flawed paper [2]. The ensuing, finally public (after three years in the secret of peer review), discussions on blogs, news commentary and follow up articles were certainly very informative on the absolute necessity of changing the ways we do science to ensure a more rapid discussion of research results [3].
One of the lessons I draw from this adventure is that the traditional publishing system is, at best ill suited (e.g. Small: three years delay), or at worst (e.g. Nature Materials) completely reluctant at considering replications or challenges to their published findings. Therefore, I am now using PrePrints (e.g. to publish a letter PNAS won’t share with their readers [4]), PubPeer and journals such as ScienceOpen where publication happens immediately and peer review follows [5].
So whilst I warmly welcome this editorial, it will need a little more to convince me that it is not a complete waste of time to use the traditional channels to open discussions of published results.
[1] The rest of letter can be found at https://raphazlab.wordpress.com/2012/12/17/letter-to-naturematerials/
[2] The article was eventually published in Small (DOI:10.1002/smll.201001465); timeline: https://raphazlab.wordpress.com/2012/12/20/stripy-timeline/
[3] https://raphazlab.wordpress.com/stripy-outside/
[4] https://raphazlab.wordpress.com/2015/11/16/pnas-your-letter-does-not-contribute-significantly-to-the-discussion-of-this-paper/
[5] https://raphazlab.wordpress.com/2015/11/17/the-spherical-nucleic-acids-mrna-detection-paradox/
Rapha-z-lab 
“Therefore, I am not using PrePrints…” should it be “now”?
LikeLike
AArgh, of course it should! Will correct. Thank you!
LikeLike
I´m just making the experience, trying to comment on a flawed paper. Very frustrating!
I really respect your persistency and your work: not stopping to raise this critical points. Thanks a lot for that.
Maybe the scientific culture will improve and implement more transparency and open, fair discussions some day. Maybe it will move some day from efficient paper/proposal production to research. Maybe it will move some day from envious competition for rare positions and reputation and from academic precarity to something better. It’s a nice idea for young scientists like me, however, right now, I’m not seeing it at all.
LikeLiked by 1 person
Thanks for your comment. My advice if you are not in a safe academic position: do what you have to do to get your concerns in the public, e.g. via PubPeer, but do not waste too much of your energy fighting the system. I hope the “nice idea” comes true but it will take time…
LikeLike
I completely agree with you, Raphaël. Although this is a welcome announcement, Nature, like nearly all journals, have a long way to go before such rosy ideals become any sort of reality. In particular, the fact that the editorial announces no specific policy or procedural changes suggests that the words will remain empty for some while to come.
Annoyingly, when I tried to point this out in the comments under the editorial my comments were removed, twice. So criticism of publications is good in principle but frowned upon in practice. I reproduce below the comment I tried to post. Like Raphaël I have a concrete and recent experience of a Nature journal NOT living up to these ideals. By a strange coincidence it was also Nature Materials (I didn’t enter into specifics in the comment).
As of writing, it seems that Nature have simply disabled the whole comment thread. Go forth and replicate – as long as it doesn’t involve the slightest criticism of Nature??
——
Journals are definitely a major obstacle to replications and corrections.
It is important to realise that the typical replication is not going to be a pleasant confirmation of the publication or, at worst, a failed replication that reveals some freak random event leaving authors and editors with their reputations unharmed. In reality, much of the non-replicating science out there arises because of problems that were apparent but ignored at the time of publication. Often a single mistake suffices to undermine the whole story, so invalidations won’t necessarily take the form of photogenic “complete” replications. If NPG are going to get serious about replications, they’ll need to deal with cases invalidating their shiny publications and damaging the reputations of the authors and editors. Journals are really bad at this and NPG are no exception, yet.
Some specific points requiring action that are mostly not addressed in the editorial:
i) robust and transparent procedures for dealing with the editorial conflict of interest;
ii) journals should consider corrections/replications for publications in other journals;
iii) invalidations should automatically be judged to have the same “impact” as the original publication;
iv) the bar for calling a result into question should not be set far higher than for the original publication;
v) give the NPG data access policy some teeth by switching to public data access.
“We welcome, and will be glad to help disseminate, results that explore the validity of key publications, including our own.” Wow! Revolutionary, if true. I’ll call your bluff. It so happens that I have a manuscript that explores the validity of a key paper in a top NPG journal. The editors won’t consider it for publication. How can you help me?
LikeLike
“Go forth and replicate”… but don’t expect to publish that in Nature!
LikeLike
I think editors should be required to address comments on published papers if they meet some simple criteria: https://elifesciences.org/content/4/e12708
LikeLiked by 1 person