post-publication peer review

A RESPONSE FROM CHAD MIRKIN’S GROUP [follow up #1/n]

Some readers might wonder why I am going on about this, so let me tell you: this is a considerably more important story than Stripy Nanoparticles Revisited. If, as I am arguing, some of this science is shaky, then it is not only the way we evaluate scientists and spend public money which are put into question, but the foundation of ongoing clinical trials.

Back to basics: in the section of Mirkin’s group PhD dissertation (previous post) that respond to our critique of their work on Spherical Nucleic Acid / SmartFlare / StickyFlare, they wrote the following:

Additionally, since the commercialization and sale of the nanoflare platform under the trade name Smartflare (Millipore), dozens of researchers around the world have participated in successful sequence-specific gene detection.[80]

Reference [80] correspond to six (half a dozen) articles, 80a to 80f (see below for details and links). Out of these six, only two are actual research papers, and, for both, the SmartFlares are a very minor addition to the work. Out of these two, only one is completely independent of Mirkin/EMD Millipore (the other one comes from Northwestern).

80a) is not primary research; it is an advertorial produced by EMD Millipore.

80b) is not primary research: it is a 300 words congress abstract (no figure). A follow up paper by the same group is discussed here.

80c) is a review and it is a collaboration between Northwestern (Mirkin’s University) and EMD Millipore. CoI statement from the paper: “D. Weldon is the R&D Manager at EMD Millipore responsible for the production of SmartFlares. Patents related to therapeutically targeting Nodal in tumor cells have been awarded to E.A. Seftor, R.E.B. Seftor, and M.J.C. Hendrix.

80d) is a research paper. It does not show in any way that SmartFlares work. It assumes it does. The SmartFlare is a minor part of the article.

80e) is not primary research: it is an advertorial in a magazine funded by company advertising (including EMD Millipore in that very issue). The author is a journalist working for the magazine, not a practicing scientist.

80f) is a research paper. It does not show in any way that SmartFlares work. It assumes it does. SmartFlares are a very minor part of the article. The authors are from Northwestern, i.e. Mirkin’s University.

 

A response from Chad Mirkin’s group

Well nearly. Possibly as close as we might get.

If you have followed the Spherical Nucleic Acid / SmartFlare / StickyFlare story on this blog, you will know that we have raised doubts about the endosomal escape of these nanoparticles which are supposed to reach the cytosol of cells where they could detect mRNAs. We have even published an article on this topic. The Mirkin group has developed the technology and it has been commercialized four years ago for the live cell detection of mRNAs by EMD Millipore.

One PNAS paper on the topic was Briley et al (see here for letter to PNAS and what happened to that letter). In his PhD dissertation (Briley, W. E. (2016), William Edward Briley respond to our criticism. I reproduce the relevant section below. One might note that there is an incredibly simple way to determine the localisation of these particles inside the cells: electron microscopy, a technique which has been used for this exact purpose for over 50 years. We have used it. The results were unambiguous.

2.3 Commentary on the Endosomal Escape of SNAs
Though the endosomal escape of SNA nanostructures such as the Nanoflare and stickyflare is evident based upon their ability to provide sequence-specific information regarding RNA levels and locations within cells, one researcher has concluded that SNAs cannot escape from endosomes.[75] That researcher [That’s me!] is ignoring the many papers now that use such architectures for sequence-specific cell-sorting experiments. Indeed, if these architectures, which are taken up by scavenger-receptor mediated endosomal pathways,[68a] do not escape the endosome, then it is difficult to understand the reports by the many groups who have documented the sequencespecific function of SNAs (all of which require endosomal escape), in antisense gene regulation,[12, 44-45, 53] siRNA gene regulation,[20, 68b, 76] nanoflare gene detection,[47, 67a, 67c, 77] and sticky-flare gene detection.[78] Perhaps the best demonstration of this sequence specificity is in the function of the multiplexed Nanoflare.[67a] This nanoconjugate, discussed in chapter 1, contains two targeting sequences specific to two different genes (actin and survivin). When cells treated with multiplexed nanoflares were subjected to siRNA that specifically knocked down the expression of survivin, a corresponding decrease of fluorescence was observed in the nanoflare’s survivin-associated fluorescence (cy5), but not the actin-associated fluorescence (cy3) compared to control.[67a] Likewise, when cells were subjected to actin-targeting siRNA, a decrease in actin-associated fluorescence was observed, with no decrease of survivin-associated fluorescence.
These results indicate that the detection by nanoflares is unique to the targeted mRNA transcript. To rule out any possible effects of the fluorophores, the Cy5 and Cy3 dyes were switched to the other gene (actin-cy5, survivin-cy3), and again the corresponding sequence-specific responses were observed. Importantly, since the development of the multiplexed nanoflare, other research groups have independently developed nanoflare-like structures capable of sequence-specific detection of 3,[57] and even 4[79] genes simultaneously. Additionally, since the commercialization and sale of the nanoflare platform under the trade name Smartflare (Millipore), dozens of researchers around the world have participated in successful sequence-specific gene detection.[80]
Further evidence of SNA endosome escape can be seen visually through analysis of the sticky-flare construct. If sticky-flares were incapable of escaping endosomes, one would expect consistent colocalization with endosomes. However, this is not the case. The patterns of β-actin targeted sticky-flares, when used in HeLa cells, instead exhibit localization around mitochondria. If such structures were limited to endosomes, it is inconceivable that they would co-localize with specific organelles. This is consistent with mitochondrial localization of multiple RNAs which
has also been identified by other groups, using multiple techniques, in HeLa cells.[74] Sticky-flare release from the endosome was further confirmed by designing sticky-flares targeting a second sequence, a U1 short nuclear RNA (snRNA). U1 snRNA is known to traffic from the cytoplasm to the nucleus. Indeed, cells treated with U1-targeting sticky-flares exhibited specific fluorescence within the nucleus. The pattern observed indicates nuclear localization through endosomal escape and sequence-specific tagging of an RNA that is actively transported into the nucleus. Again, this would not be possible if such structures were confined exclusively to endosomes. Speculation that SNAs do not escape endosomes has been fueled by the observation that the fluorescence pattern of β-actin in MEFs is punctate, which has been interpreted as an indication that the sticky-flares are simply trapped in endosomes. However, β-actin is well known to exhibit punctate fluorescence in many cases, an observation made by others in multiple cell lines, including MEFs.[81] Punctate fluorescence is very common in RNA-labeling studies and well known by researchers familiar with FISH.82 This is due to the fact that RNA is often packaged into large RNA-containing granules, which facilitates transport and translational control of the included transcripts.[82b, 83] Such packaging into granules has been extensively studied using β-actin mRNA. Thus, the fact that sticky-flares targeting β-actin were packaged into granules as previously observed, while U1-targeting sticky-flares were specific to the nucleus in the same cell line, demonstrates the functionality of the construct.
Taken together, the success of the many groups who use flare architectures for the detection and knockdown of RNA in cells, and the work of dozens of labs studying related nanoparticle constructs, provide unambiguous evidence of the ability for such architectures to escape the endosome and participate in reactions exclusive to the cytosol. The mechanism of endosomal escape for nanoparticle-based vehicles is currently unknown and is an interesting and important question that is actively being investigated by many in the field.[84]

ref2ref3

Briley, W. E. (2016). Investigation and manipulation of the local microenvironment of spherical nucleic acid nanoconjugates (Order No. 10117274). Available from ProQuest Dissertations & Theses Global. (1795522748). Retrieved from https://search.proquest.com/docview/1795522748?accountid=12117

SmartFlare controversy: independent confirmation of endosomal localization

Check this previous post for a quick summary of the SmartFlare controversy, or read all SmartFlare-related posts if you are really passionate.

At the centre of the SmartFlare controversy is the rather simple question, from an experimental point of view, of how many Spherical Nucleic Acids (to use Chad Mirkin’s terminology), if any, escape the endosomal pathway.

In contradiction with Chad Mirkin’s many peer reviewed articles and EMD Millipore marketing material, we concluded (Mason et al, 2016) that the Spherical Nucleic Acids do not escape endosomes and do not detect cytosolic mRNAs.

A few days ago, Sven Budik et al, an Austrian group published their evaluation of the SmartFlare in the context of equine embryo development. They write: “In all positive cells,
regardless of whether they occurred in equine conceptus, trophoblastic vesicle or fibroblast cell culture, the fluorescence signal showed a spotted pattern that is in accordance with the observations of Mason et al. (2016).

They also used electron microscopy to look at the intracellular localization of the particles. Here is the relevant part of their discussion and conclusion (emphasis mine):

The present study indicates that the intracellular process of nanogold particle uptake is endocytic and endosomal with a lysosomal sorting after longer incubation periods. This finding is in agreement with results from HeLa cells in vitro (Gilleron et al. 2013). Similarly, nanoparticles injected intravenously were taken up by endocytosis and later
clustered in lysosomes primarily in macrophages (Sadauskas et al. 2007). The incorporation time of lipid nanoparticlecontaining short interfering RNA gold particles in HeLa cells was similar (Gilleron et al. 2013) to that demonstrated in equine trophoblast vesicles in the present study. Accumulation of SmartFlare probes in residual bodies may be a consequence of increased stability of the immobilised oligonucleotides adjacent to the nanogold particles due to enhanced nuclease resistance (Rosi et al. 2006). In accordance with the results of Mason et al. (2016), we observed no or very few nanogold particles free in the cytoplasm, confirming a primarily endosomal and lysosomal localisation.

This observation raises the question how a specific SmartFlare probe is able to detect its target mRNA located in the cytoplasm. One possible explanation for the generation of lysosome-located specific fluorescence signals by SmartFlare probes could be the existence of specific RNA sequences imported for subsequent degradation into lysosomes (Fujiwara et al. 2013). Further studies using qRT-PCR investigating the isolated lysosomal fraction before and after incubation with specific SmartFlare probes are necessary to confirm this hypothesis. An 18S RNA nano-flare probe had a dose-dependent cytotoxic effect on porcine fetal fibroblasts (Fu et al. 2016). In contrast, no cytotoxic effects or changes in morphology after incorporation of antisense oligonucleotide nanogold particles in a mouse endothelial cell line were observed by Rosi et al. (2006). In addition, in the present study, there was no evidence that incubation with the SmartFlare probes had a toxic effect on the equine cells tested, even at higher concentrations. This is in accordance with the results of Pan et al. (2009) demonstrating that 15-nm gold particles have only low cytotoxic effects compared with the detrimental effects of small 1.4-nm gold particles.

In conclusion, SmartFlare probes pass into early equine conceptuses at stages used for embryo transfer, as well as trophoblast vesicles and cells cultured in vitro. In these early ZP equine conceptuses, the time frame (.5 and ,24 h) for SmartFlare uptake would be suitable for practical applications in commercial embryo transfer programs. Therefore, these probes are suggested to be applicable to pre-implantation genetic diagnosis before transfer of these conceptuses to the recipient.

In summary, the authors’ results are entirely consistent with our observations. They conclude, quite reasonably, that if SmartFlares detect mRNAs whilst being in endosomes, they cannot directly detect cytosolic mRNAs. This is in direct contradiction with Mirkin et al and EMD Millipore. Then, they propose that if the SmartFlares work, they maybe detect mRNAs which are in endosomes. This an interesting hypothesis that will require further study and is very different from anything published by Mirkin and EMD Millipore (the relevant reference is here). Since Budik et al do not provide any evidence that the SmartFlares actually detect mRNAs in the first place, maybe a simpler explanation is that the SmartFlares signal is unspecific and result from the probe degradation by nucleases in endosomes.

A welcome Nature Editorial

I reproduce below a comment I have left on this Nature editorial entitled “Go forth and replicate!“.

Nature Publishing Group encouragement of replications and discussions of their own published studies is a very welcome move. Seven years ago, I wrote a letter (accompanying a submission) to the Editor of Nature Materials. The last paragraph of that letter read: “The possibility of refuting existing data and theories is an important condition of progress of scientific knowledge. The high-impact publication of wrong results can have a real impact on research activities and funding priorities. There is no doubt that the series of papers revisited in this Report contribute to shape the current scientific landscape in this area of science and that their refutation will have a large impact.” [1]

The submission was “Stripy Nanoparticles Revisited” and it took three more years to publish it… in another journal; meanwhile Nature Materials continued to publish findings based on the original flawed paper [2]. The ensuing, finally public (after three years in the secret of peer review), discussions on blogs, news commentary and follow up articles were certainly very informative on the absolute necessity of changing the ways we do science to ensure a more rapid discussion of research results [3].

One of the lessons I draw from this adventure is that the traditional publishing system is, at best ill suited (e.g. Small: three years delay), or at worst (e.g. Nature Materials) completely reluctant at considering replications or challenges to their published findings. Therefore, I am now using PrePrints (e.g. to publish a letter PNAS won’t share with their readers [4]), PubPeer and journals such as ScienceOpen where publication happens immediately and peer review follows [5].

So whilst I warmly welcome this editorial, it will need a little more to convince me that it is not a complete waste of time to use the traditional channels to open discussions of published results.

[1] The rest of letter can be found at https://raphazlab.wordpress.com/2012/12/17/letter-to-naturematerials/
[2] The article was eventually published in Small (DOI:10.1002/smll.201001465

2 comments on PubPeer

); timeline: https://raphazlab.wordpress.com/2012/12/20/stripy-timeline/
[3] https://raphazlab.wordpress.com/stripy-outside/
[4] https://raphazlab.wordpress.com/2015/11/16/pnas-your-letter-does-not-contribute-significantly-to-the-discussion-of-this-paper/
[5] https://raphazlab.wordpress.com/2015/11/17/the-spherical-nucleic-acids-mrna-detection-paradox/

Nanoparticles & cell membranes: history of a (science) fiction?

One of the reason scientists, journalists and the general public are excited about nanoparticles is their supposed ability to cross biological barriers, including, the cell membrane. This could do wonders for drug delivery by bringing active molecules to the interior of the cell where they could interact with key components of the cell machinery to restore function or kill cancer cells. On the opposite side of the coin, if nanoparticles can do this, then there are enormous implications in terms of their potential toxicity and it is very urgent to investigate. But is it true? What is the evidence? How did this idea come into the scientific literature in the first place? I have been intrigued by this question for some time. It is the publication of an article about stripy nanoparticles magically crossing the cell membrane that led me to engage in what became the stripy nanoparticles controversy. It is this same vexing question that led me to question Merck/Mirkin claims about smartflare/nanoflare/stickyflare.

In the introduction of our article “The spherical nucleic acids mRNA detection paradox“, we describe the long history of the use of gold nanoparticles (“gold colloids”) in cell biology and conclude that

…, more than five decades of work has clearly established that nanoparticles enter cells by endocytotic mechanisms that result in their entrapment inside intracellular vesicles unless those nanoparticles are biological in nature and have acquired through evolution, advanced molecular tools which enable them to escape.

In the paragraph that followed, we were trying to make the point, in part using citation data of one of these 1950s pioneering articles, that this solid knowledge has been ignored in some of the thousands of recent articles on interactions of nanoparticles with membranes and cells that have appeared in the past 15 years. In his review of the first version of our article, Steve Royle criticises that latter paragraph (in his word, a “very minor” point):

I’m not a big fan of using number of Web of Science search results as an argument (Introduction). The number of papers on Gold Nanoparticles may be increasing since 2007, but then so are the number of papers on anything. It needs to be normalised to be meaningful. It’s also a shame that only 5 papers have cited Harford et al., but it’s an old paper, maybe people are citing reviews that cover this paper instead?

This is a fair point. While normalisation as well as more detailed and systematic searches might shed some light, it is rather difficult to quantify an absence of citation. Instead, I have tried to discover where the idea that nanoparticles can diffuse through membranes comes from. Here are my prime suspects (but I would be more than happy to update this post to better reflect the history of science and ideas so please leave comment, tweet, email), Andre Nel and colleagues, in Science, 3rd of February 2006, “Toxic Potential of Materials at the Nanolevel” :

“ Moreover, some nanoparticles readily travel throughout the body, deposit in target organs, penetrate cell membranes, lodge in mitochondria, and may trigger injurious responses.”

This claim is not supported by a reference, but later in the article Nel et al refer to an earlier paper entitled “Ultrafine particles cross cellular membranes by nonphagocytic mechanisms in lungs and in cultured cells” by Marianne Geiser and colleagues. These two papers, Nel et al, and, Geiser et al, have been cited respectively 5000 times and 850 times according to PubMed.

As early as 2007, Shayla Banerji and Mark Hayes had already challenged this idea of transport of nanoparticles across membranes in an elegant experimental and theoretical study which was a direct response to the two papers cited above “Examination of Nonendocytotic Bulk Transport of Nanoparticles Across Phospholipid Membranes“:

In accordance with these health concerns, Nel et al. have described some phenomena that can only potentiate fear of the negative health risks associated with nanotechnology.

[…]

Non-endocytotic transmembrane transport of large macromolecules is a significant exception to what is presently known about cell membrane permeability. Most early studies show that lipid bilayers are essentially impenetrable by molecules larger than water under physiological conditions: transport of most molecules across cell membranes is specifically cell-mediated by endocytosis.34 Endocytosis, unlike proposed passive, non-endocytotic transport, is an active cell-mediated process by which a substance gains entry into a cell. Specifically, a cell’s plasma membrane continuously invaginates to form vesicles around materials that originated outside the membrane: as the invagination continuously folds inward, the cell membrane constituents simultaneously reorganize in such a way that the material being transported into the cell is completely enclosed in a lipid bilayer, forming an endosome.35,36

[…]

The results suggest that a diffusive process of transport is not likely.

Figure 8 is particularly telling (!).

Capture

The article by Shayla Banerji and Mark Hayes has been cited 44 times.

 

Opening up peer review: the peculiar case of PNAS contributed papers

Proceedings of the National Academy of Sciences (PNAS) has two paths for submission of research articles, one standard and one less so, the famous contributed track where the submitting author has to be a member of the National Academy of Sciences of the United States of America. Peter Aldhous reviewed in 2014 this inside track and those who use it more often. He describes the contributed track as follows: “This unusual process allows authors to choose who will review their paper and how to respond to those reviewers’ comments“.

There are two elements of transparency and accountability to counterbalance this conflict of interest of having an author acting as the editor of their own article: it is recorded on the paper that it is a contributed paper, and, the names of the referees (chosen by the author) are also published. It is interesting (maybe) to note that this ancient house of PNAS has a system there which is pretty similar to what has been recently proposed as a disruptive innovation in scientific publishing by Jan Velterop and implemented by ScienceOpen, i.e. peer review by endorsement (PRO). There are a couple of differences. The first one is that PRO at ScienceOpen is open to everyone, not just National Academy members. The second one is that not just the names of the reviewers, but also the content of the reviews that is shared in PRO.

As readers of this blog will know, David Mason and I have recently challenged a particular contributed PNAS paper by the Mirkin group on StickyFlares. The discussion can be found at PubPeer (the authors did not engage). We requested some data which (after some efforts) we eventually obtained. We wrote a letter to the Editor. Which was eventually rejected by Inder Verma editor of PNAS. The letter is available at BiorXiv.

Given that neither PubPeer nor the PNAS letter to editor enabled to get any answers from the authors to our substantial criticism, we were curious to know if any of the referees had maybe raised similar issues and, if yes, how the authors had replied. Dave therefore wrote to the referees to ask whether they would share their reports. The response was negative; they could not share their reports because “referee comments from peer review have to be kept confidential as it is an essential part of maintaining the integrity of the peer review process“. I was rather surprised by this response and was moved to write the following:

01/10/2015

Dear Shana, Chris

I am Raphael; Dave is a member of my group. Apologies for pitching in and for a rather long response!

Thank you both for your replies. [paragraph edited out about the issue of whether the PNAS guidelines on choice of reviewers were followed; see the PubPeer discussion for more]

I fully share your commitment to defend the integrity of the review process, but I would urge to you to reconsider your decision to keep your comments confidential, precisely because it does not serve that very commendable aim.

It is worth considering for a moment what is the role of confidentiality and anonymity in the peer review process, and also who is in charge of guaranteeing that confidentiality and anonymity. In a traditional peer review process, the reports are confidential and anonymous: the justification is the protection of the reviewers from potential reprisals if they were to write a very critical review. The editors are in charge of protecting the confidentiality and anonymity: this is part of the contract between the editor and the reviewers. If the editor was to publish the reviews or/and, worse, reveal their identity, he/she would breach that contract and this would significantly affect the trust between future reviewers and the journal. I have myself pondered on publishing on my blog the reviews of a (rejected) submission of one of my papers, and was eventually convinced (though I am still entirely sure this was the right decision) not to do so by the detailed comments of an editor who did point out that the reviewers expected their reviews to remain confidential and that I would therefore breach their trust by doing so [1]. It is however a completely different matter for reviewers who can decide to forego their right to anonymity both immediately at the stage of the review process (“signing reviews”, usually, precisely with the motivation of increasing transparency and integrity of the review process), or later, for various reasons (nearly 200 000 examples at Publons, a site that enables and encourages reviewers to share their reports [2-3]). A recent prominent example of a reviewer sharing her reviews (on PubPeer) is Vicki Vance, who had reviewed several of the papers of Olivier Voinnet and noted serious problems (they were nevertheless published) [4]. I have never heard anyone suggesting that a reviewer who would decide to share their reviews of a paper after publication, i.e. their own scholarly evaluation of published work, would be damaging the integrity of the peer review process. I also really fail to see by what mechanism it could do so.

Obviously, the PNAS “contributed submission” path is another can of worms. Many would argue that it is in itself damaging to the integrity of the peer review process with this very unusual situation where an author chose its reviewers. In this specific case, it is hard to see any justification at all for the confidentiality of the reviews: it does not serve to protect the reviewers from potential reprisals from the author since the author has chosen its referees in the first place. The only thing it does is prevent the public (and in particular other scientists) to benefit from the insights that would be provided by sharing the reviews. I would argue that here even more than in any other case, sharing the reviews would be the best way to protect the integrity of the peer review process and therefore I hope you will reconsider,

Best wishes

Raphael

[1] https://raphazlab.wordpress.com/2013/01/03/nature-materials-peer-view-of-stripy-revisited-july-to-september-2009-confidential-or-not/ (see first comment in particular)
[2] https://publons.com/
[3] The scientists who get credit for peer review, Nature, 2014
[4] http://www.lab-times.org/editorial/e_600.lasso

Unfortunately, I did not get a reply.

The F**** word

I am talking of course of the word fraud.

It is generally understood that the f**** word is best avoided in polite company, especially when talking about the work of colleagues published in peer reviewed journals. If you really must (in which case, you’d better be critical but fair), you should instead simply point to the facts but avoid making explicitly the implication that fraud has happened (the copy and paste similarities between bands, etc; you name it).

Hopefully, from that point, journal editors and scientific institutions whose main mission is service to science and its integrity will take over and will sort out the mess.

Except that it does not happen. Here is an ordinary example:

The same authors have at least two other articles with similar problems, i.e. multiple particles from the same electron microscopy picture that look strangely similar.  Right. I am not going to beat around the bush. This is fraud. There cannot be any innocent way by which such an image can be produced. It is therefore fraud (and poor quality photoshop).

François-Xavier Coudert reported his concerns to the Editors of the respective journals. After this latest series of tweets, one editor finally responded that “authors could not provide original (primary high-res) data due to a “flood” of their lab”. End of story, says editor. Microchimica Acta will not act because they “cannot prove image was manipulated”.

The best Twitter responses so far are by Chris Waldron and Sylvain Deville

//platform.twitter.com/widgets.js

and…

//platform.twitter.com/widgets.js

Seriously though, if in a case like this, institutions and journals cannot act in a timely manner to fix the scientific record, there is no hope for cases which actually require thinking and investigations.

Here is the PubPeer thread with links to the other articles.

Update (20/12/2015): Editor of Microchimica Acta, Otto Wolfbeis, has been in touch. It is not the end of the story after all. From his email, we learn that the University of Manchester has been alerted and that a draft of a Retraction Note has been sent to the authors for comment.

Update (11/02/2016): Still no expression of concerns nor retractions… and Elsevier and Springer are still selling these fabricated articles:

Capture

Capture

Update (01/03/2016): RETRACTION of the Microchimica paper (Springer) “Following a balanced discussion of the allegations and after having consulted experts, the Editors of Microchimica Acta have come to the conclusion that there is striking evidence for manipulation.

 

 

Disclaimer: This is a personal weblog. The opinions expressed here represent my own and not those of my employer.