Stripes, open peer review and ‘copyright as a form of censorship’

This post is reproduced from a comment originally posted at PubPeer by Philip Moriarty. The title is inspired by a follow up comment by Nanonymous.

Two of the four referees who reviewed our paper for PLOS ONE have very kindly given permission to make their (anonymous) reviews publicly available. The reviews are below.

On behalf of all of the authors of the PLOS ONE paper, I’d like to say a big thank you both to the reviewers (for the considerable time and effort they put into their comprehensive reviews and for granting permission to make those reviews available online), and to PLOS ONE for contacting the referees on our behalf. (The other reviews came from Prof. Stellacci himself and another anonymous referee who raised no substantive issues with our paper (in a very short report)).

Before getting to the reviews, here’s an update on our PLOS ONE paper.

The paper was accepted for publication on August 03 2014. A suggestion of publishing a Formal Comment from Prof. Stellacci at the same time as our paper (i.e. our paper would be delayed until Prof. Stellacci’s Comment was written and peer-reviewed) was made by PLOS ONE. We rejected that suggestion because it was not in line with PLOS ONE’s guidelines on handling manuscripts which dispute published work.

To its credit, PLOS ONE accepted our arguments and did not hold us to the Formal Comment process.

…and then we hit another immensely frustrating hurdle (entirely out of PLOS ONE’s control).

PLOS ONE publishes its papers under a Creative Commons licence (to its immense credit again). This is wonderful if the paper contains only images from your own research, but in our case we are critiquing previously published work. We *have* to include images/data from previous publications.

And therein lies the rub. If we use those images/data in our PLOS ONE paper they then fall under the Creative Commons licence and can be freely used by others. This can be a problem when it comes to the copyright held by the publishers of the original (critiqued) work.

Let’s compare and contrast…

The Royal Society of Chemistry has given us (and PLOS ONE) permission to use the figures from Stellacci et al.’s work included in our paper (which were published in Chem. Comm.)

Thank you, RSC. Classy behaviour.

Wiley, however, has refused permission to include the figures from the paper in “small” (Yu and Stellacci, small 8, 3720 (2012)) which we critique.

Poor show, Wiley.

The work-around PLOS ONE has suggested is that we instead include hyperlinks to the images in question. This, of course, will result in a virtually unreadable paper and all of the hard work Julian put into generating comparisons of artefacts with Stellacci et al.’s images (in the same figure) is in many ways lost.

Deeply frustrating. But at least we’ve got the version on the arXiv. And we will upload the revised and final version of our paper there very soon.

It remains to be seen how Nature Publishing Group and the American Chemical Society will react when asked for permission to reproduce images from their journals in the PLOS ONE paper. The PLOS ONE editors contacted them, and the RSC and Wiley, on our behalf. (Thank you for this as well, PLOS ONE). NPG and ACS have not yet replied.

It beggars belief that the scientific publishing system is so screwed up that this type of farce can happen.

268 days since submission and counting.

Anyway, here are the reviews…

——————————————————————————————————————————————————-
REVIEWER #1 — First report

The manuscript by Stirling et al. addresses critically the experimental evidences presented by Stallacci et al. on striped nanoparticles up to now. In my opinion, the detail analysis presented by this manuscript overwhelmingly points to many serious and systematic experimental flaws in investigating this issue by Stallacci’s group. Therefore, I highly recommend it be accepted for publication in PLoS One.

It is very somewhat disturbing to see that how these types of experimental flaws, especially earlier STM feedback artifacts, were not addressed in earlier in the reviewing process of many high impact journals. (This, itself calls for a serious reflection of our scientific reviewing process). As far as I know, the only STM experimental evidences of the striped nanoparticle were presented by Stallacci and his collaborators. There has not been a high profile work from another totally independent group to verify its existence. Even the latest so called independent STM work from several groups resulted in a coauthored paper with Stellacci as a corresponding author.

On the other hand, the fact that this concept (especially the cartoon of these stripe particles) have been reproduced in several nanotechnology textbook, and has seemingly become a well-established scientific fact is a little disturbing to me. This calls for a serious reflection exactly how nanosceince and nanotechnology research can be conducted in more rigorous scientific manner. Of course, one problem arises from the fact there are very few groups in the world that has the capability, research interests and knowledge of both colloidal nanoparticle chemistry as well as scanning probe microscopy. On the other hand, Moriarty’s group in Nottingham is obviously capable of carrying this type of research based on their past research track record.

I do have a few suggestions that maybe the author could consider to emphasize in their revision:

1) On page 3, the author claimed “ it is clear that the stripes extend between the nanoparticles (Figure 1b). This observation alone strongly suggests that the stripes are not real surface features confined to the nanoparticles”. Although ligand interdigitation and possibly even bundling could explaining seeing feature outside nanoparticles, the orientation of the stripe could not be possibly align along the same direction (as in Figure 1b) because different crystalline facets are facing different orientations and so does the ligand molecules attached. Another issue is that with nanoparticles randomly deposited on the substrate, crystalline facets from different particles would orient randomly as well, so it is also impossible that neighboring nanoparticles have the same stripe orientation. These points need to be emphasized.

2) Regarding Janus nanoparticle formation, Figure 4 of Ong 2013 ACS Nano paper more likely is due to formation of two dimer particles. It is clear their synthesis was not generating highly uniform single sized nanoparticles. Especially for STM characterization, the sample will be washed thoroughly to remove excess ligand. This inevitably cause particle to be ligand deficient on the surface of nanoparticles. Based on our own experience, it is highly likely there is some local sintering and form nanoparticle dimer with necking formation. So any enlongated nanoparticles are primary candidates for sintering dimer rather than ligand anisotropy on a single nanoparticle.

——————————————————————————————————————————————————

REVIEWER #1 — Second Report

The revised version of Stirling et al. has made some changes to the manuscript, especially in the analysis of SANS data, but the central message of the paper remains the same. I also read the detailed response by Stering et al. to Prof. Stallacci’s referee report. To be honest, I’m a little saddened that the debate on this issue has evolved to the point that both sides are highly emotional and dwell on word-by-word trench style fighting. Scientifically, I would side with Stirling et al. that STM image from Jackson 2004 is clearly originated from instrumental artifact.

As I pointed it out in the previous referee report, the random orientation of nanoparticles on the substrate would cause stripes of nanoparticles, if it exists, to orientated randomly as well among different particles. It cannot physically orientate in the same direction, as shown in the image of Jackson 2004. This point alone would cast severe doubt on the existence of stripes. The resemblance to STM ringing artifact and the analysis done by Stirling et al. on the raw data only further confirm this point. And I think the reason this debate has waged on for so long is that people’s ego and scientific credential have been put the test here. And I think it does science no favor that we cannot admit the obvious mistake we made during the process of scientific endeavor. On the other hand, I definitely can sympathize Prof. Stallacci’s situation. I am pretty sure I wrote something wrong myself in my own publication ten years ago, and would hate to see my peers in the same field attacking it through online blogging.

Nevertheless, the manuscript by Stirling et al. presented a systematic and clear argument why data presented by Prof. Stallacci’s group lacks the conclusive evidence that stripe exist in nanoparticles. This argument deserved to be published and presented to the scientific community. Whether stripe phase exist or not obviously can not be resolved through this debate. It can only be solved through more independent research not associated with both sides of this debate.

——————————————————————————————————————————————————-
REVIEWER #3

This is a manuscript on an intensely discussed topic, the observation of “striped nanoparticles” by Stellacci and co-workers. The debate is already active for quite a while since the initial paper of Stellacci and co-workers back in 2004 (Nature Materials 3, 330), and has attracted the attention of many scientists.

The claim of Stellacci and co-workers is to have found ordered domains of molecular ligands on nanoparticles that result in stripes on the nanoparticles when imaging them with scanning tunneling microscopy (STM) (which is the method of choice since it images the particles in real space – see also comments below). While phase-separated domains have already been known, this would be the first observation of ordered domains with substantial impact on the role of nanoparticles for protein adsorption, biochemical properties of the nanoparticles, etc.

Moriarty and co-workers (and others) have criticized this work and argued that the stripes in STM images are not necessarily caused by molecular domains, but could also be a result of experimental artefacts. In the presented study, they discuss these possibilities in detail and do not only present re-analyzed original data of the Stellacci group, but also simulations with experimental settings that could easily cause similar appearances in STM images. In fact, they find that all of the STM evidence presented so far for the striped nanoparticles could also be explained by experimental artefacts. Thus, experimental evidence for the existence of stripes is still lacking, although many papers have already been published.

I fully agree with the arguments of Stirling et al. and think that they have done a very careful analysis, which should be helpful to Stellacci and co-workers to solve the problem. It should be noted that solving the problem does not mean to agree on the fact that wrong STM settings can cause problems in imaging (something that Prof Stellacci seems to agree on), but to unambiguously show that stripes nanoparticles are also imaged with correct experimental settings, something which they have to my knowledge not done. This work might be helpful in this regard since it is a very complete overview and explanation of possible STM artefacts. To my opinion it should be published as it is.
I focus in the following on the scanning tunneling microscopy (STM) results. This is because this covers the vast majority of studies assessed and is furthermore in agreement with Francesco Stellacci himself who mentions (in his response to reviewer #2) that “evidences (…) of these domains lie mainly in scanning tunneling microscopy images”. I agree with the arguments of Stirling et al. that STM is the only direct evidence for the stripes and other methods rather support this assumption than giving independent proofs. For instance, the TEM images (Fig.11) show features that are assigned to the stripe structures, but I am not convinced of this interpretation for reasons that are clearly mentioned by Stirling et al. (on page 17).

As a first observation from an STM researcher, I was astonished by two points (before going into the details of the Stirling analysis):

1) All stripes in the images point in the same direction, although different nanoparticles are involved. As far as I can see there is no reason for this and this makes an average STM scientist suspicious about the validity of the observation. Any feature which is not specific to the nanoparticles (with several ones being present in the same image), but rather follows the image orientations (related to the scan direction) is suspicious. I wonder how the image in Fig.1a of the Nature Materials 2004 publication would look if the scan direction was rotated by 45°.

2) Stripes perpendicular to the scan direction are characteristic for noise in STM images. The easiest (and very fast) way to check this is to takes another image of the same area (with exactly the same nanoparticles) and to change only the scan speed by a factor 2-5 (keeping the number of pixels, preamp gain, side length and other parameters constant). If it is noise, the periodicity will change, if it is real then it will remain. It seems that Stellacci and co-workers did not do this test because they changed several parameters at the same time, not allowing for a clear answer on this simple test. Both tests are very easy (taking only a few minutes) and I cannot find them in the work of Stellacci and co-workers.

These are only two examples, but they point in the same direction as the study by Stirling et al., namely that basic checks on image consistency are not only missing in the papers but have – as it seems – not even been carried out.

Regarding the style/format of criticism made by Moriarty and co-workers (over several years), which by the way has been (and still is) subject of debate by itself, I think that they have acted very carefully and scientifically correct. This concerns on the one hand how they criticize Stellacci’s data interpretation by using raw data and trying to re-analyze and simulate them, considering possible problems (as feedback loop artefacts etc.). To my opinion, this is the correct way if the authors themselves (who had several years for this) did not reply to the criticism by doing the same thing convincingly (see above and below for rather easy experiments that could have solved the problem very quickly in an early stage).

On the other hand, I think it is not only correct but also the best procedure (from a reader’s point of view) to criticize the work in a chronological order (as done by Stirling et al.) because many later publications are based on the earlier ones (as usually). This allows a researcher to follow the entire development and to judge it correctly. I do not agree with the comment by F. Stellacci who criticizes this practice (in his assessment) and I actually do not see any reason for being not chronological in such an assessment.

The study of J. Stirling et al. is done very carefully and with a huge effort, although it does not cover their own original work, but is devoted to assess the work of another group. This is to my opinion a merit by itself in the scientific community. The do not simply criticize the outcome of Stellacci’s work, but thoroughly discuss the data and potential experimental problems in the rather complicated setup of a scanning tunneling microscope. In particular, and importantly for the style of such a discussion, they do not accuse Prof Stellacci of intentionally modifying data or results but they raise the question whether features in the STM images (that are clearly present in the images, but maybe not real on the surface) could have been misinterpreted. Considering various potential problems in data (mis-) interpretation, their study is extremely helpful to assess the validity of the striped nanoparticles and to my opinion the outcome is very clear. In fact, their study is done in a transparent and tutorial manner, which renders it of general interest to any researcher working with STM. I would recommend it to young students in the field to get an idea of what can go wrong when doing STM measurements and what needs to be considered when analyzing unexpected features.

In this regard, I cannot support Prof Stellacci’s claim that “the vast majority of (the studies) claims are technically wrong” – I rather see the opposite and believe that Prof Stellacci tries to defend his own results and original claims rather than searching for the scientifically correct solution (by doing one of the proposed experiments or by sending the Moriarty group some of his striped nanoparticles), although the obvious facts favor the point of Moriarty and co-workers.

A very convincing argument (among many, in particular Figs.1-3) by Stirling et al is the addition of several STM images in Fig.5. It can be seen that the stripes disappear, which is a clear indication of noise rather than a physically real feature which would of course persist.

The main criticism is about the (almost famous) image from the Nature Materials 2004 work (Fig.1a there), which has been used several times and is therefore key for the entire discussion. For this reason, I think it is reasonable that Stirling et al. use it and I do not support the argument of Prof Stellacci who criticizes the dominant role of this image in the assessment. Stirling et al. nicely show how such a “striped nanoparticle” can appear in an STM image without any real stripes on the nanoparticle, simply by artefacts from the feedback loop. This is extremely convincing and I fully support their point of view.

What I find very strange is that Stellacci and co-workers could have very easily proven the validity of this image experimentally (actually a procedure that most STM users would have done immediately in the lab after obtaining such striped images – it is something like the basic rule of STM imaging) as described above by changing the scanning speed over a wide range. Stirling et al. very nicely address this point (“The stripes should be visible without ringing being present in the image”) and I fully agree with their arguments.

An important argument is that of other groups that have also seen the striped nanoparticles. Also here, I fully support the position of Moriarty and co-workers that none of these other labs could really reproduce the stripes in the same intensity and quality as Stellacci’s group. Just as an example, the de Feyter group has worked with such nanoparticles (ACS Nano 7, 8529 (2013)), but the results are not convincing at all since I can hardly see any structure that corresponds to stripes. In addition to the real space images where the claimed stripes are in the same order of magnitude as the noise level, they show an analysis of the Fourier transform where they find periodicities for both types of nanoparticles with one and with two molecular ligands – while only the latter can have them (a detailed analysis is given in Fig.10 of Stirling et al.).

The main criticism of Stirling et al. is on the data treatment by the Stellacci group, leading to experimental artefacts (as striped nanoparticles) that are then misinterpreted. Without going into all details of this very comprehensive analysis, I want to pick out two extremely important points:
1) Stellacci and co-workers used large scale images and then zoomed into a small area offline (i.e. not in the measurements but in the acquired data). This is a very unnatural experimental approach (that should not even be used in lab courses at the undergraduate level) because most experimentators would intuitively take another image with a smaller side length (but the same number of pixels) to increase the resolution in the images. Stellacci and co-workers lost pixels in their images in this way which they added artificially afterwards by interpolation, which is simply a very dangerous – if not incorrect – way of data acquisition. Accordingly, they obtain a technically impossible uncertainty of 0.026 pixels (!) in the original image. Stirling et al. have carefully analyzed this issue and I fully agree with them.
2) The current signal (Fig.1e) shows the same ripples as the stripes in the original image. However, this is a constant-current image, which should not show such structures at all. This goes together with the strange tunneling current values (discussed on page 5).

In summary, the manuscript of Stirling et al. is a very nice piece of work that was done very carefully and that must be published. In essence the Stellacci group should react on this criticism by providing convincing evidence that the stripes on the nanoparticles are real (the study of Stirling et al. is full of suggestions how this could be done) or sending their striped nanoparticles to the Moriarty lab, something that they have not done yet. I agree with Stirling et al. that this evidence is currently lacking and hope that Stellacci and co-workers could do so. The presented manuscript is an extremely important step in this (right) direction and I strongly support its publication. Its importance does not only lie in the quality of the manuscript itself, but also in the importance for the quality and style of scientific discussions on ambiguous results. I find their arguments very convincing and believe (and hope) that this work could end the long-standing debate