How do I know if an article is good? an #ACSBoston tale

This week I attended the fall meeting of the American Chemical Society in Boston. A little meeting of about 14,000 attendees. I was speaking in a symposium with an impossibly long title but which turned out to be good fun and interesting; big thanks to the organisers: Kimberly Hamad-Schifferli, Clemens Burda and Wolfgang Parak.

IMG_0117I spent most of my time in that symposium but also went to a few other sessions which tackled questions surrounding the ways we do and communicate science. I learnt a bit more about the activities of the Center for Open Science and the platform they offer to researchers to organise, plan, record and share their work (and I was even offered a T-shirt). Probably the best lecture I heard – and certainly the most entertaining – was in a science communication session “The poisoner’s guide to communicating chemistry” by Deborah Blum (now I really need to read her book).

I joined a session with the promising title of “Scientific Integrity: Can We Rely on the Published Scientific Literature?“. Judith Currano (Head of the Chemistry Library, University of Pennsylvania) discussed how to help students evaluate the quality of scientific articles; I reproduce the abstract below (italics and bold mine):

This paper, by a chemistry librarian and a professor who edits an online journal, frames the challenges facing scientists at all levels as a result of the highly variable quality of the scientific literature resulting from the introduction of a deluge of new open-access online journals, many from previously unknown publishers with highly variable standards of peer review. The problems are so pervasive that even papers submitted to well-established, legitimate journals may include citations to questionable or even frankly plagiarized sources. The authors will suggest ways in which science librarians can work with students and researchers to increase their awareness of these new threats to the integrity of the scientific literature and to increase their ability to evaluate the reliability of journals and individual articles. Traditional rules of thumb for assessing the reliability of scientific publications (peer review, publication in a journal with an established Thomson-Reuters Impact Factor, credible publisher) are more challenging to apply given the highly variable quality of many of the new open access journals, the appearance of new publishers, and the introduction of new impact metrics, some of which are interesting and useful, but others of which are based on citation patterns found in poorly described data sets or nonselective databases of articles. The authors suggest that instruction of research students in Responsible Conduct of Research be extended to include ways to evaluate the reliability of scientific information.

Now the problem of (rapidly) evaluating the reliability of an article, especially for new researchers in a particular field is a serious and acute one, so I fully approve the author’s suggestions.

However the entire paper is based largely on a false premise: the idea that it is the “introduction of a deluge of new open-access online journals” which creates this reliability problem. This is hardly the case. The difficulty in identifying poor articles is not the deluge of open access journals nor is it predatory publishing. The growth in the volume of publications is not particularly related to open access and predatory publishing can be easily identified (with a little bit of common sense and a few pointers). The abstract (and to a lesser extent the talk) also conflates the evaluation of the reliability of a journal (an impossible task if you ask me) and the reliability of an article (an extremely onerous task if you ask me, but more on this later). Do I need to comment on the “rule of thumbs“?

I do teach third year undergraduate students on a similar topic. I ask them this same question: “how can you evaluate the validity of a scientific article?”. I write their answers on the white board; in whatever order, I get: the prestige of the University/Authors/Journal, the impact factor, the quality (?) of the references… I then cross it all. I show the Arsenate DNA paper published in Science, the STAP papers published in Nature. I try to convince them that no measure of prestige can help them evaluate the quality and reliability of a paper, that the only solution they have is to read the paper carefully and critically analyse the data. If necessary, discuss it with others. If necessary ask questions to the authors.

Of course, reading carefully takes time, but there is (currently) no alternative. There is absolutely no reason to think that a paper is reliable because it is in an high impact factor journal. The Scottish philosopher David Hume (1771-1776) wrote that “A wise man proportions his belief to the evidence”…and should “always reject the greater miracle.” Many articles in high impact journals resemble such miracles and eventually turn out to be irreproducible.

The second part of my own scientific presentation focused on our ongoing SmartFlare project. On the last slide, it featured the David Hume quote as well as an updated 21st version (see below).

Capture

With the PubPeer browser extension, you can immediately see, on the journal page (or anywhere else the article is cited) if there is an existing discussion at PubPeer

There is however something simple that we can do immediately to make it easier for every body to evaluate the reliability of individual articles: sharing our critiques (positive or negative) of articles we read. If we all commit to use PubPeer and start sharing at least one review per month, this will go a very long way towards generating open discussions around articles. It will obviously not alleviate the need to read the articles and the reviews critically, but it will crowd source the evaluation and this can be very powerful (it is the model of SJS, ScienceOpen, F1000).

Capture

Nanoparticles for imaging and sensing in biology

This is the title of a 3x1H45 course which I will give early September at the European School On Nanosciences and Nanotechnologies (ESONN) in Grenoble. The focus is on inorganic nanoparticles, e.g. gold, silver, iron oxide, quantum dots. It will be the second year I give this course.

I have opted for a mostly discussion-based format centered around selected publications. Last year, I chose the publications and distributed them during the lectures, but let’s try to get more organised.

I am asking readers of this blog (optional but very much welcome!) as well as students registered for the track B of ESONN15 (mandatory: deadline Friday 28th August) to suggest at least one article for discussion (depending on the success of this crowd sourcing effort, we might or might not be able to discuss all articles). To suggest a paper, simply add a comment to this post with a reference (link to the paper would be even better).

Papers can be selected because they are historic landmarks in the field; or because they are recent ground breaking discoveries; or because they raise important questions that we need to discuss to move forward. One line justification for selecting the paper would be great.

Over to you!

Are StickyFlares smarter than SmartFlares?

Update (19/08/2015): Dave Mason has posted a detailed critique of this paper at PubPeer

A quick post before I take off to Boston tomorrow for the American Chemical Society national meeting. I informed Chad Mirkin of my Monday talk where I will discuss the SmartFlares (talk on Monday, abstract). In his reply, he pointed me to a contributed PNAS paper they published in July on StickyFlares (Links: article, Northwestern press release). The questions that this technology raises are the same as the ones raised by the SmartFlares, as discussed in a previous post. Eight years after the initial NanoFlare paper, they are still not answered in this new article.

Check the latest results of our SmartFlare studies on the open notebook and data repository.

THE SCIENCE NEWS CYCLE [2]

The paper (Nano Letters) demonstrated for the first time…

…planar undulations of composite multilink nanowire-based chains (diameter 200 nm) induced by a planar-oscillating magnetic field.

The Very Respectable Scholarly Society Press Release announced that we were moving (swimming even) towards

“…nanorobots that swim through blood to deliver drugs (video)”

Gizmag informed its readers that

“… Nanorobots wade through blood to deliver drugs”

 

The actual article says nothing about “blood” nor “drug”.

THE SCIENCE NEWS CYCLE 1 is here. This could become a regular feature. I am happy to receive suggestions via Twitter, comments below or email.

 

Nanoparticles for Cell Tracking 2015

The UK Regenerative Medicine Platform (UKRMP) Safety Hub is hosting Nanoparticles for Cell Tracking, 21-22 September 2015 at the University of Liverpool (Foresight Centre).

Date: 21-22 September 2015
Start time: 09:30
End time: 17:30
Location: The symposium will take place at the Foresight Centre, University of Liverpool, 1 Brownlow Street, Liverpool L69 3GL.

Open to:

  • Academia
  • Business/industry

Contact: For more information contact Claire Hutchinson,chutch@liv.ac.uk

The UKRMP Safety Hub was established alongside a further four Hubs to address the number of developmental challenges which need to be overcome to successfully translate promising discoveries in the field of regenerative medicine for the benefit of patients. To ensure research connects seamlessly from discovery science through to clinical and commercial application, BBSRC, EPSRC and MRC together formed the UKRMP across UK universities and research institutions. Cell tracking with nanoparticles is a major component of the Safety Hub.

The meeting will include significant time for discussions regarding the achievements, potential, and limitations of nanoparticles for cell tracking and the implications with respect to stem cell tracking in animal models and humans.

Speakers currently confirmed include:

Prof Andrew Tsourkas, Assistant Director of Program in Targeted Therapeutics, University of Pennsylvania; Developing contrast agents for molecular imaging and cell tracking applications
Prof Kostas Kostarelos, Head of Nanomedicine Lab, University of Manchester; Title TBC
Prof Quentin Pankhurst, Director of Institute of Biomedical Engineering, UCL; Biomedical Applications of Magnetic Nanoparticles
Prof Philip Blower, Chair of Imaging Chemistry, Kings College London;Cell tracking with radionuclides, both direct and reporter gene approaches
Prof Jason Davis, University of Oxford; Resonant contrast agents
Dr Tammy Kalber, Centre for Advanced Biomedical Imaging, UCL;Magnetic Targeting and multi-modal imaging
Dr James Dixon, UKRMP Acellular Hub, University of Nottingham;Enhanced delivery of functional molecules into cells
Dr Neill Liptrott, University of Liverpool; Compatibility of nanomaterials with the immunological and haematological systems
Dr Bill Shingleton, GE Healthcare; Title TBC
Dr Gabriela Juarez Martinez, Knowledge Transfer Network; Title TBC

We are accepting abstracts for both oral and poster presentations; if you would like to submit an abstract please follow the instructions on the guidance document and send to chutch@liv.ac.uk no later than30th June 2015.

Those who are successful will be notified and required to register for the meeting. Registration will open shortly after the Abstract Submission deadline.

Extracting diffusion dynamics from the fluctuations in photothermal images

 Dan-z-earth

This is a guest post by Dan Nieves, who was until recently a joint member of Raphael and Dave’s labs. Dan has moved as far as he could go from us: he is now residing in Sydney at the EMBL Australia node for Single Molecule Science at the University of New South Wales.

Today, our paper from my time at Liverpool “Photothermal Raster Image Correlation Spectroscopy (PhRICS) of gold nanoparticles in solution and on live cells was published in the new Royal Society open-access journal, Royal Society Open Science.  This journal is committed to an open peer-review system, thus, the review history and referees comments are viewable alongside the article, and also post publication peer-review in the form of a comments section below the paper is facilitated. Additionally, the data that supports the conclusions of the paper are (and have to be) readily accessible (here at Figshare). This is exciting, as not only are the discussions between authors and referees are available to everyone, but you can also join in the discussion fully after publication with access to the primary data. Therefore, the critical evaluation/re-evaluation of the work is totally encouraged and should never stop!

Our paper describes the development of an extension of photothermal heterodyne imaging; a technique used to detect and image single gold nanoparticles much smaller than the diffraction limit at high signal to noise via scattering induced by laser light absorption (nice explanation here). The extension employs fast raster-scan imaging of the sample in which fluctuations, or “streaks” (top panels, Fig.1), are observed due to the movement of nanoparticles through the detection volume during the scan. From these fluctuations it is possible to extract how fast the nanoparticles are moving from the application of image correlation analyses.  In our particular case, we applied the raster image correlation spectroscopy (RICS) method, developed in the lab of Enrico Gratton (original paper here). Briefly, after acquiring many raster scan images; the images are then spatially correlated with themselves by shifting the image pixel by pixel in all directions (x and y in this case) and calculating the correlation function.  This means repeating fluctuations within the image, i.e., nanoparticle diffusion, will be reflected in the time it takes for the spatial correlation to decay, for example, the spatial correlations for movement of slow moving objects decays quite differently to that of fast moving objects (lower panels, Fig.1). From the spatial correlations the diffusion behavior, such as the diffusion coefficient, can be extracted.  In our case, I applied the analysis to photothermal images of 8.8nm gold nanoparticles diffusing in solutions of different viscosity to verify the PhRICS approach (Fig.1). Here, we were able to extract the diffusion coefficients of the nanoparticles in the different solutions. The advantage of this approach compared to the current photothermal heterodyne techniques for probing diffusion (photothermal tracking and absorption correlation spectroscopy) is that not only can we acquire rapidly information on fast diffusion dynamics, but we can also observe the distribution of nanoparticles over the relatively large area of the image (≈ 40 μm2).

 

Fig.1

Fig.1 – Example of gold nanoparticle diffusion in solutions of different viscosity (top panel) with the corresponding spatial correlations (bottom panel).

We then turned our attention to the use of the technique to observe the diffusion of fibroblast growth factors labelled with gold nanoparticles on live cells. FGFs are involved in a wide range of essential biological processes from the formation of morphogen gradients and signalling to homeostatic control of glucose and phosphate levels.  Here, 8.8 nm gold nanoparticles were used to covalently label single fibroblast growth factor 2 proteins (FGF2-NP: via this method), and then incubated with live rat mammary fibroblast cells (Fig.2).  It was observed previously in our lab that there is significant heterogeneity in FGF2 distribution and diffusion in the pericellular matrix when bound to heparan sulphate. We found the diffusion coefficient of the FGF2-NP could be extracted, and that diffusion measurements were variable depending on the area imaged.  Additionally, it is apparent that the image data contained much more information than we could extract using the simple diffusion model applied.  The observation of the formation and dissolution of intense peaks within the images, added to the 2D-movement of such peaks from image to image (see Movie1), gives more insight into the dynamic long range restructuring of the pericellular matrix of live cells at “short” (μs and ms) and “longer” (secs and mins) timescales.

Clipboard

Fig.2 – Photothermal image of rat mammary fibroblast incubated with 600 pM of FGF2-NP.  Blue boxes indicate the areas where PhRICS imaging was performed on the cell.

Cell_1_600pMFGF2_RICS_5_STACK

Movie1 – PhRICS image series from box 5 in Fig.2

The paper is now available at the Royal Society Open Science , and if your interest has been piqued thus far, I strongly encourage you check the paper out.  Better still would be for you to engage in the post-publication comments section should you have any questions, comments or suggestions.