Merck

Do planes fly and other difficult scientific questions

The Scientist magazine reported on the ACS meeting incident. Here is Chad Mirkin’s response to their questions:

Mirkin disagrees with Levy’s assessment of the endosome entrapment. “Levy’s narcissistic approach is akin to, ‘I bought an airplane, and I can’t make it fly. Therefore, planes don’t fly, despite the fact that I see them all above me,’” he tells The Scientist.

Mirkin stresses the number of studies in which the probes have been used successfully: “There is no controversy . . . There are over 40 papers reporting the successful use of such structures, involving over 100 different researchers, spanning three different continents,” he writes to The Scientist in an email. “I think the data and widespread use of such structures speak to their reliability and utility for measuring RNA content in live cells,” he adds.

After “dishonest Rafael [sic] Levy and his band of trolls“, “scientific terrorist” and “scientific zealot“, I suppose the “narcissistad hominem, could be considered more moderate?

1920px-John_William_Waterhouse_-_Echo_and_Narcissus_-_Google_Art_Project

Echo and Narcissus, John William Waterhouse, 1903, Walker Art Gallery, Liverpool. Narcissus, too busy contemplating his image, cannot see Echo let alone planes flying above him.

As The Scientist notes, I am hardly the only one who cannot make the SmartFlare plane fly. And the plane manufacturer has stopped selling its product and does not answer questions from journalists.

Guest post: my experience with the SmartFlares, by James Elliott

CaptureThis is a guest post by James Elliott, manager of the Flow Cytometry Facility at the MRC Institute (LMS) in Hammersmith.

***

I thought it may be useful to add to the discussion about SmartFlares, their marketing and the difficulties in disseminating negative results by passing on my own experience.

We tested the system back in 2013. Sorting primary murine T cells and thymocytes on the basis of RNA expression was perhaps of most immediate interest, but of course there were countless potential applications.

The Merck Millipore rep advised us that the caveat we should be aware of in using SmartFlares was that the particles are taken up by endocytosis and that not all cells possess the machinery to allow this. Indeed, he mentioned data he had seen that only around 20% of T cells take up probes. This was puzzling as it suggested either a specific subset of endocytosis-competent cells or alternatively that uptake by T cells was broad but weak, such that only 20% of cells fell into a positive, above background gate. This in itself seemed a potentially interesting question.

To address the usefulness of SmartFlares in primary T cells (and some lymphocyte lines we had in culture) it was agreed with the rep that the sensible first step was buy positive (an ‘Uptake’ probe where fluorescence is always ‘on’ even in the absence of specific RNA) and negative (scrambled, ‘fluorescence off’) controls.

Everyone rightly comments on the extremely high price of the reagents and though we were given a discount, it remained an expensive look-see experiment.

It was useful that on the day we tried out the probes we were lucky enough to have someone with us from Merck who we like and trust to oversee what we were doing – he could vouch for the fact that we did the experiments correctly. We looked for probe uptake both on a flow cytometer and an Imagestream imaging cytometer.

Whilst we had expected lymphocytes to take up the probes poorly, in fact the big problem we had was that whilst all, or nearly all cells took up the probes, the signal from cells given the scrambled probe – notionally ‘always off’ was just as high and in most cases a little higher than that with the positive control ‘Uptake’ probe. Both showed a marked, log shift in fluorescence.

So – big problems! Why was the scrambled probe, which should have been dim or ‘fluorescence off’ giving us such a high signal? Indeed, if anything our negative control was brighter than the positive.

The rep consulted with the technical team, who were quick to point out that more meaningful comparison would have been between a scrambled and housekeeping probe (the Uptake probe merely being useful to show a qualitative result), yet this seemed to me to fudge the issue: first, surely the uptake and scrambled probes should be roughly comparable in number of molecules of fluorochrome attached or the uptake control would be of limited value – it would give a yes/no answer as to whether the cells would take up probe, but would give little clue as to efficiency. Second, the strategy of validating the system had been agreed with the rep. It was not great to then come back and say that actually this was not a good test after all. Third, and most importantly, a system in which the negative control (‘fluorescence off’!) gives a log shift in fluorescence is likely to be almost completely useless! The background would be far too high for all but the most abundant markers.

In addition to which it hardly inspired confidence that the company seemed to have validated the system very poorly – why else would they be giving a vague suggestion that maybe 20% of T cells take up probe, when in our careful (and observed) hands, they did so rather efficiently. Interestingly, in this respect I later read on a cytometry forum that, according to one US user, the company had been very up front from the beginning that primary lymphocytes don’t take up the probes. This was doubly untrue – lymphocytes do take up the probes and in the UK anyway, we were not told primary lymphocytes didn’t take up probes – the rep thought 20% of T cells did so, but was unsure about the data. Again, I was left with the impression of a poorly validated system sold by reps who were largely in the dark.

The most likely explanation for our results in follow up discussions with the company was that scrambled probe had degraded intracellularly and that this can happen in a cell type-specific way. This would mean that there would be a cell type-specific optimum time window where there was a satisfactory balance between cleavage by target RNAs and non-specific cleavage. Of course we had followed the instruction we were given at the time, but now it appeared these probably weren’t correct for our (hardly esoteric!) cells.

The suggestion was therefore that as many controls as possible would be wise.

Clearly this had become completely untenable as a system – we would have to buy hugely expensive probes and – if they worked at all, which we still didn’t know – would have to do a lot of work to establish not only the usual factors such as concentration, but also timing. And how narrow might the optimal time window be where specificity was apparent? An hour? Less? And background from non-specific signal from degrading probes would be likely to be (at least in the cells we were most interested in) a major problem for any RNA that wasn’t highly expressed.

We decided to cut our losses. I applaud those who can follow up and publish negative data that will be useful to the scientific community, but it seemed likely that for us this would end up far too expensive financially and in time and effort – quite possibly simply to show that the system might just about work, but not in any way that would be practically useful.

 

 

 

 

 

 

SmartFlare Maths

SmartFlare are nanoparticle sensors which are sold by Merck and are supposed to detect mRNA inside live cells. The technology has been developed by Chad Mirkin. In his papers, the nanoparticles are called Nano-Flares or Spherical Nucleic Acids. I am saying “supposed to” because the central question of how those sensors could possibly reach the target that they are supposed to detect has not been addressed by Mirkin nor by Merck.

After evaluating the SmartFlare, we published recently our conclusions at ScienceOpen. We ran this research as an open science project, sharing our experimental results, analyses and conclusions in quasi real time using an open science notebook. All of the imaging data can also be consulted via our online Open Microscopy Environment repository.

Gal Haimovich, who reviewed our paper, first on his blog and then at ScienceOpen, suggested we should do some SmartFlare Maths (point 4 of his list of comments). This had been at the back of my mind for some time. There are various ways to look at this problem, but all those I have tried lead to the same conclusion that the protocols, results and conclusion published do not add up. Here is what I believe the simplest way to think of the SmartFlare Maths problem. As usual, comments and corrections would be very much appreciated.

Estimation of the number of SmartFlares per cell

SF-figure adapted from Giljohann

Adapted from Giljohan et al, Figure 1b

Estimate 1. SmartFlares are added to cells at a final concentration of 0.1 nM (following Merck’s protocol). For 400,000 cells and 20 μL (following Merck’s protocol), this would result in 150,000 SmartFlares per cell, assuming that all nanoparticles are uptaken.

 

Estimate 2. Giljohann et al  (Mirkin’s group) published a quantitative study of uptake of SmartFlares in various cell lines in 2007. From their Figure 1b, we can see that in the lower concentration range tested, there is a linear correlation between SmartFlare concentration in the medium and number of particles per cell. For cells exposed to a medium concentration of 0.1 nM, this would result in an uptake of 75 000 SmartFlares per cell. In the following discussion, we will use this lower estimate. With ~50 oligo probes per SmartFlare, this would give 3,750,000 oligo probes per cell.

Oligo probes per cell versus mRNA per cell

The copy number of any specific mRNA per cell depends on sequence, cell types, signalling events etc, but typically it ranges from a few copies to a few thousands of copies. Our estimate above indicates an excess of oligo probes of at least three orders of magnitude over the most abundant mRNA.

If just 0.1% of these probes would bind their target, it would block 3,750 mRNA resulting in silencing. However, Merck and Mirkin both report that there is no silencing effect in the conditions of these experiments. It follows that more than 99.9% of the SmartFlares do not bind their target mRNA.

Fluorescence background

Seferos

Reproduced from Seferos et al, Figure 1.

Seferos et al (2007, Mirkin’s group) show that in the absence of release of the probe, fluorescence value of ~30% of the total value after release is observed (in ideal test-tube conditions, i.e. in the absence of nucleases). This is presumably due to a non-complete quenching of the fluorescence. For the SmartFlares to work, we would therefore have to detect a variation of less than 0.1% over a background of ~30%.

 

Lab Times: “Flare up over SmartFlares”

Stephen Buckingham interviewed me for Lab Times

On the face of it, Millipore’s SmartFlares are meant to be a tool cell biologists dream of – a way of measuring levels of specific RNA in real time in living cells. But does it really work? Raphaël Lévy and Gal Haimovich are in doubt.

Raphaël Lévy, Senior Lecturer in Biochemistry at the University of Liverpool, UK, was so unconvinced about SmartFlares that he decided to put the technique directly to the test (The Spherical Nucleic Acids mRNA Detection Paradox, Mason et al. ScienceOpen Research). As a result, Lévy has found himself at the centre of a row; not only over whether the technique actually does the job but as to whether it can actually work, at all – even in principle. Lab Times asked Lévy why he is in doubt that SmartFlares really work.

Lab Times:  What’s all the fuss about SmartFlares?

Read it all here (page 50-51).

I can’t resist also quoting this bit of pf the final paragraph…

In interview, Lévy is reasonable and measured in tone. But he is no stranger to controversy and can deliver fierce polemic with style.

If you have not yet, you should also check Leonid Schneider’s earlier and more complete investigation.

Don’t just take our word for it…

This is indeed a good piece of advice, offered by Merck Millipore in their leaflet entitled “It’s what’s on the inside that matters. Detect internal biomarkers in live cells.

Capture

Those three studies are not independent investigations:

Seftor et al

Capture

Lahm et al

Capture

Steve McClellan’s Nature Webinar is not a peer reviewed publication but a “webcast” for which the sponsor, no other than, well, Merck Millipore, retains full responsibility:

Capture

 

It would be nice to hear from other scientists who have bought and used the SmartFlares, especially since our results, and EMD Millipore’s own research (see Leonid Schneider’s investigation) suggest that they do not detect mRNA levels inside cell .