Guest post: my experience with the SmartFlares, by James Elliott

CaptureThis is a guest post by James Elliott, manager of the Flow Cytometry Facility at the MRC Institute (LMS) in Hammersmith.

***

I thought it may be useful to add to the discussion about SmartFlares, their marketing and the difficulties in disseminating negative results by passing on my own experience.

We tested the system back in 2013. Sorting primary murine T cells and thymocytes on the basis of RNA expression was perhaps of most immediate interest, but of course there were countless potential applications.

The Merck Millipore rep advised us that the caveat we should be aware of in using SmartFlares was that the particles are taken up by endocytosis and that not all cells possess the machinery to allow this. Indeed, he mentioned data he had seen that only around 20% of T cells take up probes. This was puzzling as it suggested either a specific subset of endocytosis-competent cells or alternatively that uptake by T cells was broad but weak, such that only 20% of cells fell into a positive, above background gate. This in itself seemed a potentially interesting question.

To address the usefulness of SmartFlares in primary T cells (and some lymphocyte lines we had in culture) it was agreed with the rep that the sensible first step was buy positive (an ‘Uptake’ probe where fluorescence is always ‘on’ even in the absence of specific RNA) and negative (scrambled, ‘fluorescence off’) controls.

Everyone rightly comments on the extremely high price of the reagents and though we were given a discount, it remained an expensive look-see experiment.

It was useful that on the day we tried out the probes we were lucky enough to have someone with us from Merck who we like and trust to oversee what we were doing – he could vouch for the fact that we did the experiments correctly. We looked for probe uptake both on a flow cytometer and an Imagestream imaging cytometer.

Whilst we had expected lymphocytes to take up the probes poorly, in fact the big problem we had was that whilst all, or nearly all cells took up the probes, the signal from cells given the scrambled probe – notionally ‘always off’ was just as high and in most cases a little higher than that with the positive control ‘Uptake’ probe. Both showed a marked, log shift in fluorescence.

So – big problems! Why was the scrambled probe, which should have been dim or ‘fluorescence off’ giving us such a high signal? Indeed, if anything our negative control was brighter than the positive.

The rep consulted with the technical team, who were quick to point out that more meaningful comparison would have been between a scrambled and housekeeping probe (the Uptake probe merely being useful to show a qualitative result), yet this seemed to me to fudge the issue: first, surely the uptake and scrambled probes should be roughly comparable in number of molecules of fluorochrome attached or the uptake control would be of limited value – it would give a yes/no answer as to whether the cells would take up probe, but would give little clue as to efficiency. Second, the strategy of validating the system had been agreed with the rep. It was not great to then come back and say that actually this was not a good test after all. Third, and most importantly, a system in which the negative control (‘fluorescence off’!) gives a log shift in fluorescence is likely to be almost completely useless! The background would be far too high for all but the most abundant markers.

In addition to which it hardly inspired confidence that the company seemed to have validated the system very poorly – why else would they be giving a vague suggestion that maybe 20% of T cells take up probe, when in our careful (and observed) hands, they did so rather efficiently. Interestingly, in this respect I later read on a cytometry forum that, according to one US user, the company had been very up front from the beginning that primary lymphocytes don’t take up the probes. This was doubly untrue – lymphocytes do take up the probes and in the UK anyway, we were not told primary lymphocytes didn’t take up probes – the rep thought 20% of T cells did so, but was unsure about the data. Again, I was left with the impression of a poorly validated system sold by reps who were largely in the dark.

The most likely explanation for our results in follow up discussions with the company was that scrambled probe had degraded intracellularly and that this can happen in a cell type-specific way. This would mean that there would be a cell type-specific optimum time window where there was a satisfactory balance between cleavage by target RNAs and non-specific cleavage. Of course we had followed the instruction we were given at the time, but now it appeared these probably weren’t correct for our (hardly esoteric!) cells.

The suggestion was therefore that as many controls as possible would be wise.

Clearly this had become completely untenable as a system – we would have to buy hugely expensive probes and – if they worked at all, which we still didn’t know – would have to do a lot of work to establish not only the usual factors such as concentration, but also timing. And how narrow might the optimal time window be where specificity was apparent? An hour? Less? And background from non-specific signal from degrading probes would be likely to be (at least in the cells we were most interested in) a major problem for any RNA that wasn’t highly expressed.

We decided to cut our losses. I applaud those who can follow up and publish negative data that will be useful to the scientific community, but it seemed likely that for us this would end up far too expensive financially and in time and effort – quite possibly simply to show that the system might just about work, but not in any way that would be practically useful.

 

 

 

 

 

 

Scientific terrorist

At the 2018 American Chemical Society National Meeting in Boston, I asked a question to Chad Mirkin after his talk on Spherical Nucleic Acids. This is what I said:

In science, we need to share the bad news as well as the good news. In your introduction you mentioned four clinical trials. One of them has reported. It showed no efficacy and Purdue Pharma which was supposed to develop the drug decided not to pursue further. You also said that 1600 forms of NanoFlares were commercially available. This is not true anymore as the distributor has pulled the product because it does not work. Finally, I have a question: what is the percentage of nanoparticles that escape the endosome.

I had written my question and I asked exactly this although not in one block as he started answering before I had made all my points. He became very angry. The exchange lasted maybe 5 minutes. Towards the end he said that no one is reading my blog (who cares), that no one agrees with me, he called me a “scientific zealot” and a “scientific terrorist”. The packed room was shell shocked. We then moved swiftly to the next talk.

Two group leaders, one from North America and the other one from Europe, came to me afterwards.

Group leader 1:

Science is ever evolving and evidenced based. The evidence is gathered by first starting to ask questions. I witnessed an interaction between two scientists. One asks his questions gracefully and one responding in a manner unbecoming of a Linus Pauling Medalist. It took courage to stand in front of a packed room of scientists and peers to ask those questions that deserved an answer in a non-aggressive manner. It took even more courage to not become reactive when the respondent is aggressive and belittling. I certainly commended Raphael Levy for how he handled the aggressive response from Chad Mirkin. Even in disagreements, you can respond in a more professional manner. Not only is name calling not appropriate, revealing the outcomes of reviewers opinions from a confidential peer-review process is unprofessional and unethical.*

Lesson learned: Hold your self to a high standard and integrity.

Group leader 2:

Many conferences suffer from interesting discussions after a talk in such way that there are questions and there are answers. So far so good. Only in rare cases, a critical mind starts a discussion, or ask questions which imply some disagreement with the presented facts. Here I was surprised how a renowned expert like Chad Mirkin got in rage by such questions of Raphael Levy and how unprofessional his reaction was. It was not science any longer, it was a personal aggression, and this raises the question why Chad Mirkin acted like this? I do not think that this strategy helps to get more acceptance by the audience. I tribute to Raphael Levy afterwards, because I think science needs critical minds, and one should not be calm because of the fear to get attacked by a speaker. Science is full of statements how well everything works, and optimism is the fuel to keep research running. There is nothing wrong with this, but definitely one also need critical questions to make progress, and what we don’t need is unprofessional behavior and discreditation.

* Group leader 1 refers here to the outcome of the reviews of this article which you can read on ChemrXiv and which was (predictably) rejected by Nature Biomedical Engineering. During the incident Chad Mirkin used these reviews to attack me.

Update: some reactions on Twitter:

“re. your exchange at if being a critical thinker is a I think this is something we should all aspire to be. Good for you.” @wilkinglab

“Do you know Rapha’s blog? Not true that no one is reading it! It is the true gem and a rare truth island!” @zk_nano

“Wow, that’s shockingly uncool.” @sean_t_barry

“What an unprofessional guy.”  @SLapointeChem

“Calling a fellow researcher a “scientific terrorist” for raising concerns and asking a question (even if you disagree with them) is shocking. Sorry to hear that there wasn’t any real discussion instead, would’ve been interesting.” @bearore

“Surprised this isn’t getting more pub. One must wonder at what point does one’s ego/reputation become more important than the science, which ABSOLUTELY must include the bad with the good.” @Ben_Jimi440

“Keep fighting the good fight tenaciously, Raphael. Like the detectives in those old film noir shows… 🤜🏼🤛🏽”  @drheaddamage

 

Multimodal cell tracking by combining gold nanorods and reporter genes (webinar)

Toni Plagge and myself did a webinar (organised by Ithera Medical) yesterday. It mostly covers these two papers:

 

References for my talk at Gold 2018

Update: the slides are available (here: gold2018-Levy2) and there is a recording of a few minutes here.

Slide 1:

Slide 2: Nanotech is bs Tweet.

Slide 3: Calling Bullshit.

Slide 4:  Dinosaur. (from here)

Slide 5: Electron microscopy of Hela cells after the ingestion of colloidal gold; C.G. Harford, A. Hamlin, and E. Parker; 1957

Slide 6: The entry and distribution of herpes virus and colloidal gold in Hela cells after contact in suspension; M. A. Epstein, K. Hummeler, and A. Berkaloff; 1963

Slide 7:

Slide 8: The spherical nucleic acid paradox; D. Mason, G. Carolan, M. Held, J. Comenge, S. Cowman, and R. Lévy; 2015

Slide 9: Excerpt from email (shared with permission).

Slide 10:  Evaluation of SmartFlare probe applicability for verification of RNAs in early equine conceptuses, equine dermal fibroblast cells and trophoblastic vesicles;  S. Budik, W. Tschulenk, S. Kummer, I. Walter, and C. Aurich; 2017

Slide 11: SmartFlares fail to reflect their target transcripts levels; M. Czarnek and J. Bereta; 2017

Slide 12: Calcium-Binding Proteins S100A8 and S100A9: Investigation of Their Immune Regulatory Effect in Myeloid Cells; J. Yang, J. Anholts, U. Kolbe, J.A. Stegehuis-Kamp, F.H.J. Claas and M. Eikmans

Slide 13: SmartFlare catalog.

Slide 14:

The conference dinner chatter way of (not) correcting the scientific record

One of the common responses of senior colleagues to my attempts to correct the scientific record goes somewhat like this:

You are giving X [leading figure in the field] too much credit anyway. We all know that there are problems with their papers. We discussed it at the latest conference with Y and Z. We just ignore this stuff and move along. Though of course X is my friend etc.

This approach is unfair, elitist and contributes to the degradation of the scientific record.

First, it is very fundamentally unfair to the many scientists who are not present at these dinner table chatters and who may believe that the accumulation of grants, prizes, and high profile papers somewhat correlate with good science. That group of scientists will include pretty much all young scientists as well as most scientists from less advantadged countries who cannot get so easily to these conferences where the truth about scientific achievements is discussed between drinks at the end of a play-acting day (for inquisitive questions at the end of talks are of course also not the right way to act).

Second, it devalues fundamentally the role of the scientific record. We are basically accepting that it does not matter whether what gets published is right or wrong.  Here, I’ll insert an anecdote. I reviewed, a couple of years ago, a high profile review authored by a senior colleague in the field of nanoparticles. One of the figures highlighted a paper which I knew to be fundamentally wrong. In my review, I pointed that fact. The senior colleague did not dispute that the paper was flawed but he opted for keeping the figure, not discussing the fact the paper was wrong. His (post-modern) argument was that the  “concept” was important.

Dinner chatter is fine. But please also share your criticisms, e.g. via PubPeer.

(and by the way, if you could comment on our preprint on Spherical Nucleic Acids, that would be much appreciated)

 

The great answer to people saying that #preprints are not peer-reviewed

That perfect title is courtesy of (see tweet below)

On Monday (25/06), we will publish a preprint about the spherical nucleic acid technology. Our paper was prompted by the publication in Nature Biomedical Engineering of “Abnormal scar identification with spherical-nucleic-acid technology” by Yeo et al.

The great answer is… review them! I issued a call to review our preprint before it comes out and I have now sent the article to a number of colleagues across the world. I am very much looking forward to their comments good or bad. The comments will be posted on PubPeer. If you have some time on your hands this Friday or over the weekend to look at the paper, drop me an email and I will also send you a preview copy.

Yeo et al corresponding authors were provided with a copy of our preprint two weeks ago but unfortunately they have not responded. I hope they will post comments on PubPeer. We are planning to subsequently submit a version (hopefully improved thanks to the comments) to Nature Biomedical Engineering. It is however sometime rather difficult to debate the scientific literature through the official channels of traditional journals so this route via preprint will accelerate this important discussion.