The stripy controversy as a window into the scientific process*

* h/t Retraction Watch for the title

Stripy Nanoparticles Revisited was published a year ago.

Science is self correcting? [part one: doing the right thing]

At the core of the scientific process as we imagine it, there is the principle that science is self-correcting. Errors are made, but then, if they have any bearing on future studies, they get corrected. That principle failed (and continues to fail) dramatically. Consider the following.

The first “stripy nanoparticles” article was published in 2004 in Nature Materials. The flaws in that article were many and obvious but colleagues most qualified to identify those flaws, i.e. SPM experts, were not interested enough to react: they privately laughed but publicly did nothing. Correcting the flaws in the literature was “somebody else’s problem“.

In 2005, a graduate student in Francesco Stellacci’s lab at MIT also recognized that the observed stripes were a scanning artifact. He did control experiments and showed that the same stripes occur in the absence of particles on the same instrument with the same settings. He also did simulations that explain how those patterns can be generated by the STM feedback system.  At that point, retracting the Nature Materials paper would have been “doing the right thing“. It was not retracted.

In 2006, the first article was followed  by a second article (in J Am Chem Soc), which acknowledges the presence of the artifact, but desperately tries to demonstrate that the features on the particles are the “reality” (with no less than 18 figures). The article is surreal – it makes the case that the artifact is present everywhere except on the particles, where those stripes which look just like the artifact, are instead, the “reality”. An extraordinary case of confirmation bias.

What happened in 2006 is the very exact opposite of self correction. Predrag’s demonstration was simple and clear but remained confined within MIT [it only became public when Predrag contacted me after the publication of our article last year]. Instead, in the official scientific record, no doubt were raised and an impressive pile of stripy figures and articles started to accumulate. That pile continues to grow at an impressive rate, even after the publication of our article. The count is now at about 35 articles from the same group, with more striking examples of confirmation bias.

Science is self-correcting? [part two: crossing the line]

Universities, journals and funding agencies have proclaimed ethical rules which thankfully generally align. One of those shared rules is that re-use of text or figures (even your own text or figures) is not allowed without proper attribution. If this rule is broken, you would therefore expect rapid action to correct the scientific record.

The stripy controversy includes several examples of data re-use and it is informative to look at how editors and institutions have responded to those cases and what have been the final (?) outcome.

Nature Materials was informed as early as 2009 of this issue since my letter to the editor accompanying our submission did mention that “The same image of the same nanoparticle is used in both the ChemComm and the Nature Materials articles.” That did not lead to any action (and our submission to Nature Materials was rejected)At the end of 2012, with the increased scrutiny brought by the controversy, several other data re-use cases were discovered. Editors of the corresponding journals were contacted by Dave Fernig. MIT and EPFL were also informed of these concerns as well as of other concerns regarding data sharing. The initial responses from Editors are reported by Dave here. Eventually, corrections were issued at both PNAS and Nature Materials. Both of these journals are members of the Committee on Publication Ethics (COPE).

The Journal of Scanning Probe Microscopy however is not a member of COPE and its Editor clearly indicated (within minutes of Dave’s email) that he had no intention to enforce any ethical policies. More worryingly maybe, the EPFL investigation has closed and while it has been successful in ensuring compliance of Francesco Stellacci with his obligation of sharing data, it has not led to the correction of the scientific record in the case of the Journal of Scanning Probe Microscopy article. Three figures of that article contain image reuse (from two different articles) without attribution and the analyses presented in other figures correspond to data from yet a third article. The extent of data re-use in this article would probably warrant a full retraction in a journal that follows COPE guidelines. However, the paper is not retracted; instead is still cited in all recent publications as evidence of thorough and rigorous statistical analysis of the stripes [more on this here for the interested reader].

I guess that the conclusion for this part is somewhat nuanced: the system kind of worked for PNAS and Nature Materials; the editors eventually did the right thing and followed COPE guidelines. The system did not work for the more obscure Journal of Scanning Probe Microscopy and EPFL/MIT/COPE rules were not enforced (that may still happen, or maybe there is an explication/justification for these instances of data re-use in which case I will happily publish a correction to that post and other relevant posts as soon as it is communicated to me). As an aside, I note that the Royal Society of Chemistry is a member of COPE and that therefore Nanoscale Editor-in-Chief (European office) is in charge of applying COPE guidelines to Nanoscale authors.

Science is self-correcting? [part 3: critiquing is not nice]

For science to be self-correcting, there should be an incentive for authors to publish failed replication studies and disseminate analyses which identify fundamental flaws in the peer reviewed literature. Quite clearly there is currently no such incentive. I do sometimes get  asked why I engaged in this controversy. I have no personal conflict with Francesco Stellacci. I had not met him when I submitted a technical comment to Science, and, I had still not met him when I submitted the first version of Stripy Nanoparticles Revisited to Nature Materials. My reason to engage was simple: I had identified flaws in articles which had a significant impact on my field of research and I concluded that it was important to correct the record so that we could build on more solid foundations. I was conscious it would be difficult but I certainly had not predicted it would take three and a half years to publish our manuscript. I also had not predicted the extravagant events of the past 12 months, e.g. Predrag contacting me, a Nature Editor trolling my blog anonymously, the data re-use mentioned above, etc.

We need a system where confrontation of ideas is encouraged and is part of the norm, not one where it takes enormous amounts of effort and unreasonable length of time to publish a “revisited” paper. We need a system where engagement in scientific discussions is rewarded, not one where criticism (online or through the peer-reviewed literature) is seen as not nice “because we are all humans“. The latter quote is a reference to the now famous ACS Nano editorial (excellent responses at Chembark, Nature and Pubpeer). Various reasons for this specific editorial at this specific time have been proposed, from ongoing discussions at PubPeer of papers from one of the ACS Nano Editors to the uncovering by bloggers of a recent cases of fraud. Paul Weiss chief-editor of ACS Nano has denied that any single blogger were targeted but, like Paul Bracher, I certainly took it as a personal attack, especially since the latest stripy article was published in the same journal issue.

The “critiquing is not nice” line is not  very different from the initial response to Monica Byrne naming Boraz as her harasser.  This is inappropriate, not nice. A standard response to expressions of concerns about sexism or racism: we can’t talk about this, it is too serious and too damaging (to the reputation of the offender). There is one additional reason to stay silent. This is from Priya Shetty’s (excellent) piece about sexual harassment and the “deafening silence” that surrounds it:

Other than in whispered conversations in coffee shops or quiet meeting rooms, I’ve never known any woman to name the man involved, for the same reason that most women don’t speak up about sexual harassment in the workplace in general – science and journalism are both fields in which funding and job opportunities are increasingly precarious, and women stay silent for fear that their careers will suffer.

The same peer pressure applies in science around cases of bad science or misconduct and the grey area in between.

Conclusion?

None. You write it.

About these ads

8 thoughts on “The stripy controversy as a window into the scientific process*

  1. “A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.”

    This is a famous quote by Max Planck, who probably, in his frustration, would have shared your view that science is not self-correcting, at least not fast. The stripy particle story shows indeed that consensus in the so-called scientific community is at least as political as it is rational but I would maintain that ultimately science does correct itself, at least in those cases that matter. If one day the stripes are needed and cannot be made, this will happen. If not, it remains like the much discussed sound of the tree that falls in the forrest when nobody is there to listen. The very question loses its relevance. The other problem with the stripes is of course that their existence cannot be falsified, which, if we apply Popper’s criteria, places the whole finding outside science. This may well be part of the problem. Who wants to slay a unicorn? Curious to see what FakeRapha will make of this.

    • It is theoretically possible to disprove the stripes. For this one would need good molecular contrast on the head-groups to clearly define their positions. On a curved surface this is very hard, yet not definitely impossible.

  2. It is a shame that most scientists (I was one of them!!) just laughed at the 2004 Nature Materials paper but did nothing in public. As Brust pointed out, the community is as political as it is rational.

  3. I think there is a much bigger issue here. Back in the day you would have the PI in the lab doing experiments and expert in the experiments he was publishing on. He/she was close to the discovery process. Now with big prof’s having “supergroups” the PI has no practical expertise of the experiments that all his/her postdocs are performing. They tell him “we’ve discovered this” and the PI has no expertise to objectively criticise the work. Magnify this with the PI’s pressure on the postdocs to produce beautiful data to be published in high impact journals and you have a recipe for disaster. I have a feeling that in the future a lot of these Supergroups papers are going to be torn to shreds. I for one find it very strange that some of these supergroups only publish in high impact journals. Does everything always work?

  4. The line that got me in that awful ACS Nano editorial was “we still should not consider each publication from a competitor as being potentially wrong”.

    Yes we should! We should be firm in the knowledge that everything in the literature, including our own work, is potentially wrong. How can science function without viewing everything through the lens that it is potentially wrong. If we take the view that peer-reviewed work is not potentially wrong, then we are saying it is definitely correct. How then can science be self correcting?

  5. @Julian Stirling, indeed. To state that what is written is the truth and cannot be questioned means that we have moved from science to religious belief. This is the realm of theology and metaphysics, not science. As regards “proof”, there is currently a singular lack of evidence for phase separation of ligands on nanoparticles into stripes. A lack of evidence does not mean that it is impossible that this may happen with some combination of nanoparticle, ligands and solvent. However, it is the lack of evidence and the source of what is claimed to be evidence in the gold nanoparticle system this is the subject of discussion.
    @Mathias Brust, though stripes are not “needed” now, the properties of gold nanoparticles claimed to be imparted by stripes are needed urgently. In this respect, I note that the first claims for stripes were published in 2004, yet there has not been a single animal study, let alone a clinical trial since. The “need” appears to remain unmet.

  6. ferniglab,
    There is a giant graveyard of discoveries hyped with the “may lead to new drugs” pitch that have never accomplished anything of note. Even if they had gotten to an animal study or clincal trial, the most likely outcome would be that they would fail there.

    Not sure if you are a part of the chemistry department, but ask anyone who is to think back to any speaker that might have given a talk about some new molecular scale thing (nanoparticles, *RNA, bucky balls, triazole rings of some sort etc.) and how promising (in virto, animal tumour models etc.) their data looked. You can even filter for the kind of thing that people in the audience seemed genuinely impressed/excited about.

    What was the impact of any of those discoveries 10 years later? Could it be that everyone is well aware that everyone has to oversell to meet the requirements of getting grants funded?

    • @ nanonymous
      I am not a chemist, but I have often heard such facts. The important thing to consider is that if something looks promising people will probably try it. If it doesn’t work they probably won’t publish their findings and other people will try again. How many researcher hours have been spent so far by people trying to make stripy nano-particles generate these properties?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s