peer review

Time to reclaim the values of science

This post is dedicated to Paul Picard, my grand dad, who was the oldest reader of my blog. He was 17 (and Jewish) in 1939 so he did not get the chance to go to University. He passed away on the first of October 2016. More on his life here (in French) and some of his paintings (and several that he inspired to his grandchildren and great-grandchildren). The header of my blog is from a painting he did for me

A few recent events of vastly different importance eventually triggered this post.

A  (non-scientist) friend asked my expert opinion about a campaign by a French environmental NGO seeking to  raise money to challenge the use of nanoparticles such as E171 in foods. E171 receives episodic alarmist coverage, some of which were debunked by Andrew Maynard in 2014. The present campaign key dramatic science quote “avec le dioxyde de titane, on se retrouve dans la même situation qu’avec l’amiante il y a 40 ans {with titanium dioxide, we are in the same situation than we were with asbestos 40 years ago}” is from Professor Jürg Tschopp. It comes from an old media interview (2011, RTS) that followed a publication in PNAS. We cannot ask Professor Tschopp what he thinks of the use of this 5 years old quote: unfortunately he died shortly after the PNAS publication. The interpretation of this article has been questioned since: it seems likely that the observed toxicity was due to endotoxin contamination rather than the nanomaterials themselves. There is on the topic of nanoparticles a high level of misinformation and fear that finds its origins (in part) in how the scientific enterprise is run today. Incentives are to publish dramatic results in high impact factor journals which lead many scientists to vastly exaggerate both the risks and the potential of their nanomaterials of choice. The result is that we build myths instead of solid reproducible foundations, we spread disproportionate fears and hopes instead of sharing questions and knowledge. When it comes to E171 additives in foods, the consequences of basing decisions on flawed evidence are limited. After all, even if the campaign is successful, it will only result in M&M’s not being quite as shiny.

I have been worried for some time that the crisis of the scientific enterprise illustrated by this anecdote may affect the confidence of the public in science. In a way, it should; the problems are real, lead to a waste of public money, and, they slow down progress. In another way, technological (including healthcare) progress based on scientific findings has been phenomenal and there are so many critical issues where expertise and evidence are needed to face pressing humanities’ problems that such a loss of confidence would have grave detrimental effects. Last week, in the Spectator, Donna Laframboise published an article entitled “How many scientific papers just aren’t true? Enough that basing government policy on ‘peer-reviewed studies’ isn’t all it’s cracked up to be“. The article starts by a rather typical and justified critique of peer review, citing (peer-reviewed) evidence, and then, moves swiftly to climate change seeking to undermine the enormous solid body of work on man-made climate change. It just happens that Donna Laframboise is working for “a think-tank that has become the UK’s most prominent source of climate-change denial“.

One of the Brexit leaders famously declared that “people in this country have had enough of experts”. A conservative MP declared on Twitter that he”Personally, never thought of academics as ‘experts’. No experience of the real world. Yesterday, Donald Trump, a climate change denier was elected president of the USA: “The stakes for the United States, and the world, are enormous” (Michael Greshko writing for the National Geographic). These are attacks not just on experts, but on knowledge itself, and, the attacks extends to other values dear to science and encapsulated in the “Principle of the Universality of Science“:

Implementation of the Principle of the Universality of Science is fundamental to scientific progress. This Principle embodies freedom of movement, association, expression and communication for scientists, as well as equitable access to data, information and research materials. These freedoms are highly valued by the scientific community and generally well accepted by governments and policy makers. Hence, scientists are normally able to travel to international meetings, associate with colleagues and freely express their opinions regardless of factors such as ethnic origin, religion, citizenship, language, political stance, gender, sex or age. However, this is not always the case and so it is important to have mechanisms in place at the local, national and international levels to monitor compliance with this principle and intervene when breaches occur. The International Council for Science (ICSU) and its global network of Members provide one such mechanism to which individual scientists can turn for assistance. The Principle of the Universality of Science focuses on scientific rights and freedoms but implicit in these are a number of responsibilities. Individual scientists have a responsibility to conduct their work with honesty, integrity, openness and respect, and a collective responsibility to maximize the benefit and minimize the misuse of science for society as a whole. Balancing freedoms and responsibilities is not always a straightforward process. For example, openness and sharing of data and materials may be in conflict with a scientist’s desire to maintain a competitive edge or an employer’s requirements for protecting intellectual property. In some situations, for example during wars, or in specific areas of research, such as development of global surveillance technologies, the appropriate balance between freedoms and responsibilities can be extremely difficult to define and maintain. The benefits of science for human well-being and development are widely accepted. The increased average human lifespan in most parts of the world over the past century can be attributed, more or less directly, to scientific progress. At the same time, it has to be acknowledged that technologies arising from science can inadvertently have adverse effects on people and the environment. Moreover, the deliberate misuse of science can potentially have catastrophic effects. There is an increasing recognition by the scientific community that it needs to more fully engage societal stakeholders in explaining, developing and implementing research agendas. A central aspect of ensuring the freedoms of scientists and the longer term future of science is not only conducting science responsibly but being able to publicly demonstrate that science is being conducted responsibly. Individual scientists, their associated institutions, employers, funders and representative bodies, such as ICSU, have a shared role in both protecting the freedoms and propagating the responsibilities of scientists. This is a role that needs to be explicitly acknowledged and embraced. It is likely to be an increasingly demanding role in the future.

It is urgent that we, scientists, reclaim these values of humanity, integrity and openness and make them central (and visibly so) in our Universities. To ensure this transformation occurs, we must act individually and as groups so that scientists are evaluated on their application of these principles. The absurd publication system where we (the taxpayer) pay millions of £$€ to commercial publishers to share hide results that we (scientists) have acquired, evaluated and edited must end. There are some very encouraging and inspiring open science moves coming from the EU which aim explicitely at making “research more open, global, collaborative, creative and closer to society“. We must embrace and amplify these moves in our Universities. And, as many, e.g. @sazzels19 and @Stephen_curry have said, now more than ever, we need to do public engagement work, not with an advertising aim, but with a truly humanist agenda of encouraging curiosity, critical thinking, debates around technological progress and the wonders of the world.

 

A welcome Nature Editorial

I reproduce below a comment I have left on this Nature editorial entitled “Go forth and replicate!“.

Nature Publishing Group encouragement of replications and discussions of their own published studies is a very welcome move. Seven years ago, I wrote a letter (accompanying a submission) to the Editor of Nature Materials. The last paragraph of that letter read: “The possibility of refuting existing data and theories is an important condition of progress of scientific knowledge. The high-impact publication of wrong results can have a real impact on research activities and funding priorities. There is no doubt that the series of papers revisited in this Report contribute to shape the current scientific landscape in this area of science and that their refutation will have a large impact.” [1]

The submission was “Stripy Nanoparticles Revisited” and it took three more years to publish it… in another journal; meanwhile Nature Materials continued to publish findings based on the original flawed paper [2]. The ensuing, finally public (after three years in the secret of peer review), discussions on blogs, news commentary and follow up articles were certainly very informative on the absolute necessity of changing the ways we do science to ensure a more rapid discussion of research results [3].

One of the lessons I draw from this adventure is that the traditional publishing system is, at best ill suited (e.g. Small: three years delay), or at worst (e.g. Nature Materials) completely reluctant at considering replications or challenges to their published findings. Therefore, I am now using PrePrints (e.g. to publish a letter PNAS won’t share with their readers [4]), PubPeer and journals such as ScienceOpen where publication happens immediately and peer review follows [5].

So whilst I warmly welcome this editorial, it will need a little more to convince me that it is not a complete waste of time to use the traditional channels to open discussions of published results.

[1] The rest of letter can be found at https://raphazlab.wordpress.com/2012/12/17/letter-to-naturematerials/
[2] The article was eventually published in Small (DOI:10.1002/smll.201001465

2 comments on PubPeer

); timeline: https://raphazlab.wordpress.com/2012/12/20/stripy-timeline/
[3] https://raphazlab.wordpress.com/stripy-outside/
[4] https://raphazlab.wordpress.com/2015/11/16/pnas-your-letter-does-not-contribute-significantly-to-the-discussion-of-this-paper/
[5] https://raphazlab.wordpress.com/2015/11/17/the-spherical-nucleic-acids-mrna-detection-paradox/

Opening up peer review: the peculiar case of PNAS contributed papers

Proceedings of the National Academy of Sciences (PNAS) has two paths for submission of research articles, one standard and one less so, the famous contributed track where the submitting author has to be a member of the National Academy of Sciences of the United States of America. Peter Aldhous reviewed in 2014 this inside track and those who use it more often. He describes the contributed track as follows: “This unusual process allows authors to choose who will review their paper and how to respond to those reviewers’ comments“.

There are two elements of transparency and accountability to counterbalance this conflict of interest of having an author acting as the editor of their own article: it is recorded on the paper that it is a contributed paper, and, the names of the referees (chosen by the author) are also published. It is interesting (maybe) to note that this ancient house of PNAS has a system there which is pretty similar to what has been recently proposed as a disruptive innovation in scientific publishing by Jan Velterop and implemented by ScienceOpen, i.e. peer review by endorsement (PRO). There are a couple of differences. The first one is that PRO at ScienceOpen is open to everyone, not just National Academy members. The second one is that not just the names of the reviewers, but also the content of the reviews that is shared in PRO.

As readers of this blog will know, David Mason and I have recently challenged a particular contributed PNAS paper by the Mirkin group on StickyFlares. The discussion can be found at PubPeer (the authors did not engage). We requested some data which (after some efforts) we eventually obtained. We wrote a letter to the Editor. Which was eventually rejected by Inder Verma editor of PNAS. The letter is available at BiorXiv.

Given that neither PubPeer nor the PNAS letter to editor enabled to get any answers from the authors to our substantial criticism, we were curious to know if any of the referees had maybe raised similar issues and, if yes, how the authors had replied. Dave therefore wrote to the referees to ask whether they would share their reports. The response was negative; they could not share their reports because “referee comments from peer review have to be kept confidential as it is an essential part of maintaining the integrity of the peer review process“. I was rather surprised by this response and was moved to write the following:

01/10/2015

Dear Shana, Chris

I am Raphael; Dave is a member of my group. Apologies for pitching in and for a rather long response!

Thank you both for your replies. [paragraph edited out about the issue of whether the PNAS guidelines on choice of reviewers were followed; see the PubPeer discussion for more]

I fully share your commitment to defend the integrity of the review process, but I would urge to you to reconsider your decision to keep your comments confidential, precisely because it does not serve that very commendable aim.

It is worth considering for a moment what is the role of confidentiality and anonymity in the peer review process, and also who is in charge of guaranteeing that confidentiality and anonymity. In a traditional peer review process, the reports are confidential and anonymous: the justification is the protection of the reviewers from potential reprisals if they were to write a very critical review. The editors are in charge of protecting the confidentiality and anonymity: this is part of the contract between the editor and the reviewers. If the editor was to publish the reviews or/and, worse, reveal their identity, he/she would breach that contract and this would significantly affect the trust between future reviewers and the journal. I have myself pondered on publishing on my blog the reviews of a (rejected) submission of one of my papers, and was eventually convinced (though I am still entirely sure this was the right decision) not to do so by the detailed comments of an editor who did point out that the reviewers expected their reviews to remain confidential and that I would therefore breach their trust by doing so [1]. It is however a completely different matter for reviewers who can decide to forego their right to anonymity both immediately at the stage of the review process (“signing reviews”, usually, precisely with the motivation of increasing transparency and integrity of the review process), or later, for various reasons (nearly 200 000 examples at Publons, a site that enables and encourages reviewers to share their reports [2-3]). A recent prominent example of a reviewer sharing her reviews (on PubPeer) is Vicki Vance, who had reviewed several of the papers of Olivier Voinnet and noted serious problems (they were nevertheless published) [4]. I have never heard anyone suggesting that a reviewer who would decide to share their reviews of a paper after publication, i.e. their own scholarly evaluation of published work, would be damaging the integrity of the peer review process. I also really fail to see by what mechanism it could do so.

Obviously, the PNAS “contributed submission” path is another can of worms. Many would argue that it is in itself damaging to the integrity of the peer review process with this very unusual situation where an author chose its reviewers. In this specific case, it is hard to see any justification at all for the confidentiality of the reviews: it does not serve to protect the reviewers from potential reprisals from the author since the author has chosen its referees in the first place. The only thing it does is prevent the public (and in particular other scientists) to benefit from the insights that would be provided by sharing the reviews. I would argue that here even more than in any other case, sharing the reviews would be the best way to protect the integrity of the peer review process and therefore I hope you will reconsider,

Best wishes

Raphael

[1] https://raphazlab.wordpress.com/2013/01/03/nature-materials-peer-view-of-stripy-revisited-july-to-september-2009-confidential-or-not/ (see first comment in particular)
[2] https://publons.com/
[3] The scientists who get credit for peer review, Nature, 2014
[4] http://www.lab-times.org/editorial/e_600.lasso

Unfortunately, I did not get a reply.

We are anonymous (or not). We need you to join. We are (mostly) making scientific discussion in the open possible and easy.

The last thing you probably want to read is one more article about anonymity in (post-publication) peer review. The topic has been covered recently by Bastian, Blatt, Lawrence, Oransky, Moriarty & PubPeerNeuroskeptic, Schneider to name just a few. I am sorry. I’ll keep it short.

I have decided to sign the peer review reports I write as a referee. Yet, I insist that attacks against anonymity in post-publication peer review are unfair, misguided and counterproductive. These two positions might seem contradictory. Bear with me to the end of the post and, hopefully, you might agree they are not.

Social media (and in particular PubPeer) have played a role in pretty much all recent scientific controversies, in part because the traditional channels are at best inefficient and at worst useless. Journal editors and some authors do not know how to react when criticism of articles appear on these platforms. One possible reaction is to shoot the messenger. If it is anonymous, call them anonymous cowards and question the motives. If it is not, call it cyber bullying.

Philip Moriarty, a strong supporter of PubPeer, has nevertheless titled his blog contribution to this debate “We are anonymous. We are legion. We are (mostly) harmful.” With friends like this, who needs enemies… I, and others, have responded in the comments section.

We can argue all of 2016, but the key practical question are the following. Your colleague/student has read a paper and has some interesting comments that she would like to share with the world;

Say you have chosen the first option and your colleague/student has now posted her critique on PubPeer (anonymously). You happen to also know the authors.

These are very simple questions. How we answer them has implications. Peer review is central to our practice, yet publicly engaging in scientific discussion on someone else’s work is often seen as not nice. This is the cultural barrier that we need to break. Many authors make the choice of not responding to carefully crafted criticism of their work (whether the criticism is anonymous or not). A colleague recently contacted me querying my opinion on a paper. We exchanged a few emails. We agreed there were several problems and possible errors of interpretation. I suggested to share our critique on PubPeer thereby giving the original authors the opportunity of a reply (and the rest of the community the opportunity to contribute to the discussion). He replied (SIC, smilies included): “Sorry, not for me 🙂 I do not like sharing 🙂“. This is the culture we have to change. We did not even get to discuss the possibility of anonymity.

Leonid Schneider argues that most of PubPeer is not really post publication peer review. Instead, it is calling out fraud. While anonymity is OK when you call out fraud, it would problematic in cases of scientific arguments. Even if we accepted the latter (which I do not), the distinction is artificial. There is a continuum of practices from outright fabrication to cherry picking of data and extremely optimistic interpretation of results (Twitter convo on this point here), e.g. it is common that extensive statistical analysis of data is necessary to demonstrate fraud. Any analysis of published data is post-publication peer review, whether it results in new hypotheses, questions and clarifications, or suspicions of fraud.

I disagree that anonymity is a problem and I can see plenty of valid, honest, reasonable (and even anodyne) reasons why you might choose to comment anonymously. If you insist it is a problem, fine, but I hope you will agree that it is secondary to getting valuable critiques of published work in the open. And, most importantly, do not side with Blatt who is calling for authors not to respond to criticism of their work.

It is sometimes argued that anonymity in peer review is fine because there is someone, the editor, who knows the identity of the reviewer. The power asymmetry, and therefore the potential for abuse, is however much larger in formal peer review than in post-publication peer review. At PubPeer, an anonymous comment stands purely on its merits; a scientifically strong response from the authors will bring credit to the authors. In formal peer review, the anonymous referee comes with the prestigious vetting of the editor. Furthermore, editors who are often not experts and always stretched with time, rely largely on these reports for their decisions; those have an impact on career progression, eventually grant funding, etc. The accountability of reviewers is little to none. I have started to sign my reviews during the course of 2015 with some hesitations, but my commitment is now firm. I have had some positive feedback and in one case (where I had rejected the paper), an email from an author asking me for further advice on their revised ms before submitting elsewhere. For more reasons to sign your reviews, check this.

So, to conclude, here is my advice for 2016: contribute to PubPeer (anonymously, or not, I care little), sign your peer reviews… and publish them when possible.

 

 

 

 

 

A good day for science; respect to the Editor…

Earlier, I reported on the publication of our article on the internalisation of peptide-capped nanoparticles in cells. Today, I want to share with you the publication process as it happened at PloS One. The paper was submitted on the 20th of November 2014. The academic editor sent his decision, major revision, along with two referees reports on the 22nd of December, i.e. one month after submission [great turn around time!].

Reviewer 2 was very supportive but reviewer 1 much less so: there appeared to be a real difference of interpretation regarding the impact of cell-penetrating peptides on the intracellular localisation of ingested nanoparticles. The reviewer also requested additional experiments that we could not easily do at this time and that we felt were unnecessary to support our main conclusions. The academic editor himself, Dr Pedro V. Baptista [more on PloS One editorial process here], was author on a paper which in some ways could be seen as conflicting with our results and interpretation. The response to the referees and editors took me a long time to write. It was submitted on the 29th of January. I share it below.

The paper was accepted on the 6th of February. I welcome this decision, not just because our paper gets published -this is of course also great news!-, but because it demonstrates that there is space for open scientific debate in the peer reviewed literature. For this, I am immensely grateful to Dr Baptista.


Response to the referees.

Dear Dr Pedro V. Baptista

On behalf of my co-authors, I would like to thank you for handling our article and to thank the reviewers for their careful reading and for their comments.

Reviewer 2 notes that the context of our ms is the existence of conflicting reports on the effect of TAT and HA2 on intracellular fate of nanoparticles. Indeed, some articles have reported efficient access to the cytosol, while other studies indicate that most particles remain confined in endosomal compartments. Our own experiments are in line with this second group of articles. Reviewer 2 notes that “the study is well designed and executed and the results are interpreted appropriately”. Reviewer 2 supports publication in its current form.

Reviewer 1 has concerns about novelty. Reviewer 1 also suggests that we should add three references. These fall in the first category mentioned above, i.e. articles that support the notion that TAT enables access to the cytosol. It is of course appropriate that we should cite studies from both groups of articles. One of the three, […], was in fact already cited. We have now added the other two, i.e.: […]

Experiments related to this topic have led to many articles in the past 10 years. However, the persistence of conflicting reports and the importance of the topic for many envisioned applications require new insights. This we have provided through the use of imaging modalities that provide information across different scales: electron microscopy measures what occurs to a few nanoparticles in a very small part of the cell; photothermal microscopy measures what happens to the bulk of nanoparticles across a large part of the cell. This combination is thus uniquely able to address, in at least one cell type and a particular formulation of nanoparticle, the fate of TAT-functionalised nanoparticles after they bind to the cell surface.

Below we respond to the detailed queries of reviewer 1 and trust that the manuscript now meets the standards required for publication in PLOS One.

Dr Raphaël Lévy, rapha@liverpool.ac.uk

Detailed response to reviewer 1 queries:
• Novelty. Our article is a significant piece of work that adds useful information towards understanding and clarifying the impact of cell penetrating peptides on intracellular localisation of nanoparticles. The work is novel because it builds on a new imaging methodology that directly images the nanoparticle cores (as opposed to an attached fluorescent molecule) and gives a better overview of an entire cell than just electron microscopy. It is also novel because our peptide self-assembled monolayer approach enables us to do systematic variations of the surface chemistry of the nanoconjugates.
• “To include as a new figure, the extinction spectra of all the nanoconjugates as well as all the scattering spectra […]”. The reviewer is right that extinction spectra are very useful to characterise functionalisation and colloidal stability. We have added the requested figure as Fig. S0. For the conjugates used in Fig. 1, the formation of the self-assembled monolayers results in a minimal shift of the plasmon band of ~1-3 nm. This shift is small compared to the width of the plasmon peak. Because photothermal microscopy relies on absorption at the wavelength of our heating laser which matches the position of the maximal absorbance, differences due to a 1-3 nm plasmon shift are negligible. Interestingly, particles presenting a higher percentage of TAT in their monolayer do show a larger plasmon shift indicative of aggregation. We have modified the paragraph on the formation of the SAMs as follows: “Formation of the monolayer was immediately visible because of the increased colloidal stability and of a small red shift of the nanoparticles plasmon band (Fig. S0). Higher proportions of TAT in the monolayer resulted in nanoparticle aggregation and therefore were not used for further studies (Fig. S0).”

• “To include the images and quantification in Figure 1 with cells only with naked gold nanoparticles and cells only with PEG-gold nanoparticles and compare intensities.” The images and quantification for “cells only” were already included (Fig. 1A and first column of Fig. 1F). We have not included “naked gold”. Instead, as a reference point, we have used PEG-gold particles that have a capping composition made of CALNN and CCALNN-PEG. “naked gold” does not remain naked: non-specific adsorption of proteins, e.g. serum albumin in the cell medium, very rapidly changes the properties of the surface [see for example, Time Evolution of the Nanoparticle Protein Corona, Casals et al., ACS Nano, 2010, 4, pp 3623–3632]. The CALNN and CCALNNPEG composition was optimised, as discussed p 7, line 213-220 and Fig. S2 “Gold nanoparticles uptake decreases with increasing percentages of CCALNN-PEG”. The selected composition leads to minimal uptake as shown in Fig. 1B and the second column in Fig. 1F. From this reference composition, we have made systematic variations where we include defined percentages of the two functional peptides (dHA2 and TAT). For all these conditions, exemplary images are shown in Fig. 1 A-E, additional images are shared via figshare (http://dx.doi.org/10.6084/m9.figshare.1088379) and the quantifications are shown in Fig. 1F.

• “To perform other technique to quantify the gold content […].To include more time points in the TEM studies […]. […] the efficacy results reported by the authors are premature without the additional data described above.” While we agree that the suggested experiments are interesting, they are not necessary to reach the conclusions arrived at in the ms. Those conclusions do not concern “efficacy”, but increased uptake and intracellular localisation. The increase in photothermal signal as well as in the counts of nanoparticles in EM images unambiguously demonstrate increased uptake. The non-homogenous distribution of signal observed in the photothermal images and the electron microscopy analyses unambiguously rule out cytosolic distribution of the nanoparticles. The time point of 3 hours used in our studies is a key point both from the perspective of applications and of cell entry mechanisms. We agree that a systematic analysis as a function of time after uptake would provide further insights into endocytotic mechanisms, but it is outside of the focus of this study. Furthermore, it has been done extensively by cell biologists since the 1950s using a variety of probes. Notably, one of the first applications of gold nanoparticles in biology precisely focused on the mechanisms by which cells probe their external environment (Electron microscopy of HeLa cells after ingestion of colloidal gold, Harford et al., J Biophys Biochem Cytol 1957 3:749-756; reference added into the ms).

The standards in the field have been to publish only one or two representative electron
microscopy images. The photothermal imaging provides a unique means for the reader to understand nanoparticle distribution over biologically representative scales. Importantly, we are sharing here 942 EM images and 37 photothermal images. By publishing all of our data alongside the study [1], we enable other scientists to check and challenge our conclusions and propose alternative hypotheses. PLoS One is a particularly good forum for our article because of its commenting platform where this discussion can continue in the open after the publication of the article.
[1]. DOIs of the data:

10.6084/m9.figshare.1088379, 10.6084/m9.figshare.875584, 10.6084/m9.figshare.875630, 10.6084/m9.figshare.875545, 10.6084/m9.figshare.875477, 10.6084/m9.figshare.874219, 10.6084/m9.figshare.874153, 10.6084/m9.figshare.874033, 10.6084/m9.figshare.873852, 10.6084/m9.figshare.1088399, 10.6084/m9.figshare.1246458, 10.6084/m9.figshare.1246609,
10.6084/m9.figshare.1246622, 10.6084/m9.figshare.1246660, 10.6084/m9.figshare.1246696, 10.6084/m9.figshare.1246707

Novelty, reproducibility, and data sharing in (nano)materials science

Half-random ranty post that might develop into something more structured at some point… Feedback very much welcome.

Andrew Maynard has blogged about the extent to which novelty should (or, in fact, should not) be the main consideration for the evaluation of nanomaterials risks (initially published as an editorial in Nature Nanotechnology). It’s entitled “Is novelty in nanomaterials overrated when it comes to risks” and is well worth reading in full. A central point is that:

Novelty as a result is a subjective, transient, and consequently a rather unreliable indicator of potential risk. It tends to obscure the reality that conventional behaviour can sometimes lead to harm, and that mundane risks are still risks. And it favours the interesting (and possibly the headline-grabbing) over the important. But if novelty is an unreliable guide to potential risk, how can approaches be developed that help identify, understand and manage plausible risks associated with emerging materials and the products that use them?

Apparently unrelated (but wait for the next paragraphs), there are various initiatives to encourage or even mandate sharing of data related to the characterization of (nano)materials. It is thought that this will boost innovation and facilitate the coming together of computational and experimental work. Maybe the most impressive and concerted effort comes from the White House Office for Science and Technology as exemplified by this post It’s Time to Open Materials Science Data. Publishers have smelled something and are moving to the area of providing services for data sharing and curation; NPG launched Scientific Data in partnership with FigShare; Elsevier has just launched an initiative specifically targeted to open data in Materials Science.

Now for the (arguably subtle and tenuous) link. Novelty is overrated not just when it comes to risk. It is overrated in materials science full stop. This seems not intuitive; surely scientific endeavour in materials science is about discovering new materials. The problem here (and arguably the opportunity too) is that there is an immense combinatorial space of potential new materials. We work on peptide-capped gold nanoparticles. By varying the peptide sequences and making various mixed monolayers, we can potentially generate hundreds of novel materials every day (and we do make a fair number). The combinatorial space of potential nanomaterials vastly exceed the number of potential molecules. Most of these materials are not interesting, but they are novel: nobody made them before.

I see a lot of research articles which can be summarised as

  1. This is a novel nanomaterial (and it truly is: nobody has made before this gold-nanorod-with-carbon-dots-at-the-tips-graphene-oxide-on-the-side-and-some-antibody-labelled-conductive-polymer-wrapped-around [1])
  2. It could be used for [delete as appropriate] energy/biological imaging/curing cancer (and it will never be).

When it comes to safety, Andrew argues convincingly that the focus should be on plausible scenarios rather than on novelty. When it comes to what should be curiosity-driven science, there seems to be a lot of new materials generated for the sole purpose of highly improbable applications rather than in the pursuit of general principles that would help us explore the materials landscape. This has the very unfortunate consequence that the materials characterisation is often poor and limited to whatever is thought to enable the envisioned application. An extremely large proportion of these new materials are made by a single group for the purpose of a single paper. The experiments are not reproduced independently. Capturing all of this data into platforms that are open and suitable for data mining is a noble and worthwhile purpose which I support, but it must be accompanied by a change of focus and higher standards of characterisation otherwise I fear that it will not help understanding much.

Thanks to who chronicled the reaction of materials scientists to an OFST presentation at the MRS conference in Boston in December 2014.

[1] Novel Nano-Lychees for Theranostics of Cancer; Charles Spencer and Edna Purviance; Nature Matters-to-all (2015) 7  101-114

Identity theft: a new low in the stripy nanoparticles controversy

If you are familiar with the controversy and wonder why a PloS One comment thread disappeared today, skip directly to recent events.


Jackson et al, Nature Materials 2004

Jackson et al, Nature Materials, published 18 April 2004

In 2004, Alicia Jackson (now deputy director of the Biological Technologies office at DARPA), Jacob Myerson and Francesco Stellacci (now Constellium Professor at EPFL) mis-interpreted a common scanning probe microscopy artefact for a real structure. They submitted their results to Nature Materials, which published the work. This error was to become the starting point of a controversy which, more than 10 years later, is still unraveling.

Predrag Djuranovic

Predrag Djuranovic was a PhD student in Stellacci’s group at MIT

In 2005, concerns were raised by Predrag Djuranovic, then a student in Francesco Stellacci’s group. According to Lauren Wolf writing for Chemical & Engineering News last year:

he discussed his concerns with his department head at MIT […]. A few months later, an official investigation of Stellacci launched at MIT and Djuranovic left Stellacci’s research group. MIT closed its review three years later, in 2008, finding Stellacci not guilty of academic misconduct. The university did, however, suggest that Stellacci should do additional work to substantiate his 2004 findings.”

MIT's Building 10 and Great Dome; adapted from this  pic

MIT’s Building 10 and Great Dome; adapted from this pic

MIT was not convinced by the published data but instead of sharing those concerns with the scientific community, they kept them secret and Stellacci was asked to “substantiate” what he had published. This –trying to substantiate a structure which had been postulated to explain an experimental artefact– is what he has now been doing for more than 10 years… with over 30 articles published.

stripy nanoparticles revisited

Stripy Nanoparticles Revisited, published 23 Nov 2012

The controversy entered the public domain in 2012 with the publication of our article entitled Stripy Nanoparticles Revisited in Small: I was not aware of Predrag’s work but had come to the same conclusion. Francesco Stellacci published his response alongside our paper. He, and co-authors, also published several other articles around the same period. At the end of last year, Julian Stirling, and co-authors, published in PloS One an extensive Critical Assessment of the Evidence for Striped Nanoparticles which demonstrates that there is no evidence for the existence of the stripy nanoparticles. A response from Francesco Stellacci is still expected at PloS One.

Scientists interested in the details of the scientific arguments can refer to the articles linked to above (and references therein) as well as the blog posts and PubPeer discussions.

I wish this was all about science and differences of interpretation. I have touched previously on the re-use of figures in different articles. This eventually led to two corrections, one at Nature Materials and one at PNAS.  An EPFL investigation was triggered, opened, and, eventually, following the report of an international independent panel, closed. A third article, pretty much entirely based on data re-use, has not been corrected.

I decided to start blogging about the controversy after the publication of our article in 2012. I used the blog to present our paper, discuss the response (without having to wait three years), publish an invited post from Predrad Djuranovic, other experts (Philip Moriarty, Mathias Brust, and Quanmin Guo), and many other things . Other blogs and the post-publication peer review platform PubPeer also became important fora.

Unfortunately, just as the traditional literature has its flaws and can be gamed by scientists who do not respect the rules, so do these new modes of scientific communication. I have argued earlier that the stripy controversy was a window into the scientific process revealing the serious failures of self correction mechanisms. It is also a window into the frightening flaws of online post-publication peer review – I say this as a strong and determined supporter of post-publication peer review.

Those who believe in stripes have never engaged openly in the online debate (they have even vigorously complained that it was taking place, likening it to cyberbullying). Instead, we’ve had, in chronological order:

  • a troll posting over a period of two days a large number of comments on this blog… he eventually was identified as an Editor of the journal.
  • A fakerapha blog (that still exists) and a fakerapha twitter (the account has been removed) who impersonated me on the Chembark blog.
  • A text book case of Gish Gallop by “unreg” (a debating technique named after, and often used by, creationists) at PubPeer, It is to their immense credit that Julian Stirling, Philip Moriarty and several anonymous peers have engaged, repeatedly and honestly with “unreg”… until it became absolutely beyond doubt that this was no ordinary – if a little heated – discussion. Maybe best to let Nanonymous, an anonymous but insightful and regular commentator of the controversy describe that thread:

    Wow, the same thing again and again, in spite of being addressed again and again.

    and later

    […] it is interesting to reflect on the motivations for comments like these. They seem to come from an identity that either: 1)Does not understand basic scientific reasoning.2)Is vested in obfuscating some fairly straightforward arguments against clear and obvious errors in scientific work.Perhaps both. It could be just some random troll, but clear evidence that there are people acting in very bad faith out there.   Nanonymous

  • Possibly that thread also includes sockpuppetry with “unreg” and “ProfSTM/BioNanoChair” being the same person.

These things happened on blogs or at PubPeer, i.e. platforms which accept anonymous writing. It is hard to think of how they could be completely avoided while retaining anonymous comments.  The discussions of the costs/benefits of anonymous post-publication peer review is ongoing (I agree with Peer 1 in that thread).

In the last few weeks, the stripy nanoparticles discussion has moved to PloS One which enables comments directly on the article. Plos One has a clear policy requiring contributors to “unambiguously identify themselves with their first and last names, their geographic location, […] Any registered user who is found to have provided false name or location information will have their account suspended and any postings deleted.” In spite of this:

  • A “Gustav Dhror” posted over 10 comments (Gish Gallop again) about “Problems with Fig 4”. There is simply no doubt that Gustav is not a real person but instead a sockpuppet. There is no person with that name who has ever published any article on anything, let alone STM, nanoparticles or surface science. Even first year PhD students get their names listed on some website in academic institutions but  “Gustav Dhror” draws zero hits on Google, or more precisely two: the PloS One thread and a baby name website. After I posted a comment (and contacted PloS One) raising the issue of false identity, that thread stopped… but a few days later…
  • A “Dr Wei Chen” started another thread entitled “Multiple flaws with this paper“. The user profile linked to a real and existing Wei Chen, Professor at Suzhou Institute of Nano-Tech and Nano-Bionics in China. Given the context and previous events, I eventually took the liberty of calling Wei Chen to check whether he was “Wei Chen”, commenter at PloS One. And he was not. He had no idea. He had never posted anything at PloS One. Someone had stolen his identity to post these comments. Someone who wanted to discredit our paper (with flawed pseudo-scientific arguments), was not prepared to do it under their own name, but was prepare to steal someone else’s identity to do it.

PloS One has been contacted and has now removed the “Dr Wei Chen” thread. They have been in contact with “Gustav Dhror” and should post a statement regarding that thread tomorrow. In the paragraphs above, I have used links to WebArchives where you can see how the pages looked like this morning.