In a Times Higher Education article two weeks ago, Paul Jump discussed the current legal threats on post-publication peer review highlighted by “the case of Fazlul Sarkar, a distinguished professor in cancer research at Wayne State University in Detroit [who] claims that anonymous comments posted on PubPeer this summer led to the withdrawal of a $350,000 (£220,000) a year job offer by the University of Mississippi.”
Rebecca Lawrence, Managing director 0f F1000 Research, responded to the above article with a letter entitled “An anonymity problem“, suggesting that anonymous commenting was not appropriate when “scientists’ livelihoods are at stake because of competition for funding and jobs“. Similarly, in an article at The Conversation, Andy Tattersall, Information Specialist at University of Sheffield, presents the fact post-publication peer review may have an impact on scientists as a potential cause for concern.
It should not be a cause for concern. It is normal (but not the norm) that what we publish (rather than the impact factor of the journal in which it is published) and how we respond to critiques of our work, should have an impact on our careers.
The problems in Sarkar’s papers are numerous. The fact that some scientists respond to reasonable criticism of their published work with abuse, legal threats or lawsuits are a clear demonstration of why anonymity is in some cases necessary. Are Rebecca and Andy really suggesting that scientists should not be accountable for what they publish?
Rapha-z-lab 
“When scientists’ livelihoods are at stake because of competition for funding and jobs, a process that allows anonymity would seem to provide an open invitation for inappropriate behaviour by some.”
An anonymity problem indeed. Rebecca Lawrence doesn’t seem to understand that it is trivial for people to communicate anonymously, PubPeer just makes the process easier. PubPeer in some way shape or form will always be there, it is ridiculous to insinuate that it is a “problem”. If people opt for the anonymous option, why not facilitate that to at least see what the end users of science want? I think I could take the above quote and apply to grant application reviews as well…
FWIW I don’t find F1000 a compelling product. Many years ago when F1000 was new, I recall my adviser asking his students to essentially write astrotruf for buddy papers. I can’t even easily log in to F1000 to check what I wrote is there or even credited properly (I I can’t find a registration link 2 clicks and after suffering a bunch of marketing nonsense). I much prefer PubPeer.
I thought this article described what I guessed F1000 to be like:
The discussion on PubPeer focuses on critical/constructive analysis while my experience and the link above suggests that “the editorial policy of F1000 Research incentivizes short, positive comments”. I much prefer the former.
LikeLike
Thanks for this. I’m not sure where I said in my article that scientists should not be accountable for what they publish, everyone should be accountable for anything they publish online.
There are two ways of looking at this, anonymity as a barrier and facilitator in that some will not use an anonymous system as they don’t know who is having the conversation and the possibility that the anonymity allows for more bad behaviour as people sit behind their disguise – just like 4Chan or YouTube comments. They have used anonymous peer-review traditionally as that is the system, the post-publication review is very optional at the moment.
It could also be a facilitator that allows less confident scientists to discuss and comment without the fear that more senior colleagues come along and potentially humiliate them. I am in huge favour of using every technology available to forward science – but we have to think about whether peer review in its previous guise has worked and whether we can take those lessons and shape them into a better system for communicating research. The problem will be that commenting and reviews are conducted by humans and history has taught us that some humans just aren’t very nice 😉
LikeLike
Andy, there is something about your tone here I find presumptuous.
You villify 4chan, no doubt there are all sorts of sundry things being discussed on that board…but probably no different than discussions that are happening on the street or in people’s living rooms. You seem to want to limit “bad behavior” by limiting anonymity.
People can behave badly anonymous or not, but luckily we believe in consistent reasoning across human beings making it relatively easy to suss out the intellectual merit of a given piece of writing regardless of who wrote it.
There also seems to be a conflation with “confidence” and “seniority”. It isn’t about confidence, it is that many scientists are well aware of “bad behavior” by colleagues when their results are questioned.
LikeLike
I agree that these conversations happen wherever – but the social web has lead to a massive growth in online communication – everyone has always had an opinion on something – the Web gives a greater soapbox platform than we had before. I want to limit bad behaviour period, but am not against anonymity – I haven’t said that, but am raising potential issues with that model, like open comment has its pros and cons.
You say:
“we believe in consistent reasoning across human beings making it relatively easy to suss out the intellectual merit of a given piece of writing regardless of who wrote it.”
But is that always the case? many ideas and beliefs that are made public, whether that be a couple of people or over the Web can be open to ridicule. At the time that person may be right, but the Web is a great conduit for spreading opinions and news fast – so untruths as well as the truth can be spread in hours, by then some damage may be done. On the flipside – how long did Andrew Wakefield’s MMR paper exist in print before being retracted from The Lancet? In my opinion not all academics are reasonable people, they are no different from other sections of society – academia like other professions suffers from peer pressure, bullying and suppression. From my experience, and we can only speak of experiences, there are issues relating to confidence across the fields of research, early career researchers can feel a subconscious pressure to get their heads down and get on with publishing work, anything else could be deemed as excess to requirements. You are talking about scientists, I’m thinking of researchers as a whole.
LikeLike
Thanks Andy for your comment.
You have not said that scientists should not be accountable, but you have presented the fact that scientists may loose their jobs or grants because of anonymous comments as a potential problem, with a reference to Sarkar’s case (and another one to an unspecified case).
It would only be a problem if 1) those concerns highlighted in PubPeer have no ground or, 2) if scientists should not be accountable for what they publish.
If, as seems to be the case, we agree that scientists should indeed be accountable, cases like Sarkar et al would only be adequate examples of concerns with anonymous #pppr if they were fundamentally unfair. A summary look at the PubPeer link however will reveal a number of cases of gels manipulated and reused for different experimental conditions.
Raphael
LikeLike
I have been following this debate for a while, and it gives the uncomfortable impression that nothing is more serious to us scientists than our careers. Should not the science be more important? Imagine an actor, rock star or opera singer suing a magazine or website for publishing anonymous criticism of their performance. That would be laughable. Sing better next time, would be the reply. Should I be made liable for the losses of a film company when I tell all my friends not to see a certain film because I did not like it? Why should scientists be protected by law against public criticism? Is it fair? Perhaps not, but we are accountable to those who allow us to do what we like and what we should do as well as we can. Unlike artists, we at least have a way to defend ourselves against unfair criticism since there are clear and non-debatable criteria against which our work can be assessed. This should be the way to respond to criticism, not a law suit.
LikeLike
One only has to recall the “vigorous” debate at the Royal Society that followed the presentation of Darwin’s paper. Imagine if Huxley had been silenced through the courts… …doubtless many would wish that he had.
As for anonymity, it is there to protect the weak again the powerful; the powerful being largely those who are career, rather than science oriented. Questioning is critical to science, otherwise we have religion, with a caste of high priests and a nice career ladder from the novice to the high priest(ess). My personal preference is Huxley, by a distance that is approximately the diameter of the universe and the explicit provision of anonymity, to encourage questioning.
LikeLike
Thanks for this. I would clarify that I am certainly not suggesting that scientists should not be accountable for what they publish; quite the opposite in fact. I am saying that scientists, whether they are the authors of the original work or they comment on others’ work, should be completely accountable for what they say. If there is a genuine problem with a piece of published work then the scientists who are highlighting this should be willing to publicly stand by their concerns. The problem as I see it (and this is a general problem, irrespective of the details of this particular case) is that anonymity enables researchers to say things without such accountability and, particularly where the stakes are high, this can on occasion lead to inappropriate behavior which is not good for science as a whole.
LikeLike
There are now a large number of cases where concerns reported anonymously on PubPeer have led to corrections in the scientific records, including a number of retractions. Whatever the motivations of the individuals for posting those comments, and for remaining anonymous, they have made a very important contribution to scientific progress. They may have saved millions of $ of public money by preventing spending based on flawed data. They may have saved PhD students from spending three years of their life trying to reproduce flawed work before leaving science utterly disappointed. They may even have saved life if the work was of biomedical relevance leading to clinical trials.
Criticism has to stand on its merits. Authors of papers are accountable for their publications, but if someone has identified fraud or flaws and want to share this information anonymously, it is extremely beneficial to society if that can be done easily. If the information is wrong, it can be challenged and dismissed. There are perfectly respectable reasons why someone might prefer anonymity, e.g. the work you would like to criticize has been authored by someone who is editor in the journal where you intend to submit your next article, or she is in charge of millions of dollars in a funding body where you intend to submit your next grant, or he is the buddy of your PhD supervisor.
I was dismayed by your letter to THE because it did sound as an attack on PubPeer at a time where a scientist is trying to use the law against PubPeer to silence criticism of their work. At such a time, I would only expect solidarity from scientists and everyone involved in the publishing industry.
I am very curious to what will be your attitude when someone anonymous will identify serious flaws in an article published at F1000 Research and will publish their comments on PubPeer. This will no doubt happen one day. Will you dismiss this criticism as not worthy of consideration, or will you thank the anonymous contributor and investigate?
LikeLike
Say you are applying for a job, and someone lets your potential employer know about your poor standards, would your employer not have the right to re-evaluate your application based on that information?
This whole episode reads like: “I stood to get a big, big (!) paycheque. But someone informed my new employer that I’m actually not worth it. And now I have to make do with my normal income.”
What scientist is worth seven others, by the by? I can imagine one being a bit better than another one, but seven times better? Or am I taking this too personally? Are salaries in the US all like this?
B.
LikeLike
@ F1000 people
Your error is to assume good faith. There are people who are considered very successful scientists, at the pinnacle of the career structure, who have cheated, and the evidence was published. The stakes are really high if you decide to call them out. It’s not a question of a polite scientific discussion that everybody “should” of course be happy to engage in. You will point to evidence that they cheated. In turn, you risk not “humiliation”, but having your career sunk without compunction: it is you or them, after all. In those situations there is just no middle ground. A comment web site can’t please everybody at once and has to make a choice. Whether a site allows anonymity makes all the difference. It turns out that PubPeer and Retraction Watch facilitate commenting, F1000 facilitates controlling the comments.
As you based your article partly on the Sarkar case, why don’t you give us a few examples of specific “innapropriate” comments? Did you read any before voicing your opinion?
LikeLike
I sense the problem is that we are conflating two related but distinct issues, one being peer review and the other being research misconduct. My point was a much more general one (and not really specific to this case) which was that the purpose of peer review and scientific critique is not to criticize the authors, but to assess the content of the article so as to ensure that the science is rigorous and enough detail is given for it to be replicated by others. If the criticism is constructive and objective and focused on the research, peer reviewers should be able to stand up for their views.
Of course if there are concerns about possible scientific misconduct (deliberate fabrication, falsification, or plagiarism), most good journals (including ourselves) adhere to the COPE (Committee on Publication Ethics) guidelines to deal with such accusations; these guidelines are specifically designed to correct the scientific literature. Such publishers have processes in place to work with institutions, who are ultimately responsible for their employees and should have their own processes in place to respond to whistle-blowers and deal with cases of alleged scientific misconduct, where anonymity for the whistle-blower may indeed be appropriate.
LikeLike
@ Rebecca and Andy
Speaking of conflation, it is true that you conflated anonymous scientific comment with the worst internet trolling, apparently as part of an F1000 marketing ploy. In doing so, you ignored the fact that in a scientific context anonymous comments are only damaging if they have substance. And if they have substance, what is inappropriate about them? “Those two published figures are very similar” is something that every reader can verify (as could you if you read the comments you imply are inappropriate – do remind us which ones you read before writing your articles). The readers can and will draw their own conclusions. That’s not the same at all as an unfair restaurant critique on Yelp or other trolling, which readers have no way of verifying.
On F1000, you accept “peer review” but nothing that could be considered indicative of misconduct. So authors careless, stupid or dishonest enough to publish “suggestive data” are protected from (anonymous) comment on that aspect of their publication. The result is that readers are only informed about small problems but not about big ones. That does offer some comfort to the (minority of?) people who made some innocent mistake. But the much delayed communication of serious doubts to working scientists is a real problem. There’s a balance to strike and I’m unconvinced that F1000 have it right.
Yes, many journals say (even believe) that they adhere to the COPE guidelines. But the reality is that the process is slow, inefficient and error prone. Journals and institutions have very obvious conflicts of interest. There are many, many examples of good journals never quite getting around to doing the right thing or talking themselves out of robust action.
LikeLike
@irritatedresearcher
“Speaking of conflation, it is true that you conflated anonymous scientific comment with the worst internet trolling, apparently as part of an F1000 marketing ploy. In doing so, you ignored the fact that in a scientific context anonymous comments are only damaging if they have substance. And if they have substance, what is inappropriate about them?”
Just to clarify that I didn’t treat anonymous scientific commenting as the same as the worst internet trolling, but I think it does open the door to it. I’ve seen what I would regard as troll behaviour by academics in person and over the Web – such as discussion forums. Certainly nothing I have written is a marketing ploy for F1000 – although it’s obvious to see I am interested in all new forms of scholarly communication and keen to explore the pitfalls and merits. I am not dismissing anonymous or named commenting, but we have to see how things have played out on the Web with regards to both forms of feedback (non academic) as such as initiatives such as the Khan Academy being a stepping stone to the more formalised university MOOCs.
I would like to see all forms of scholarly communication as one size does not fit all, I think there are so many issues relating to both, agenda, ethics, privacy and etiquette and the simple issue that comments can be mis-understood via phone or online text as we don’t hear the tone or see the person.
Hopefully open discussion on this topic can only help, I want more open/anonymous online discussion and debate but feel the lines are blurred somewhat and that’s no bad thing for the minority of academics actively participating online, when the rest eventually join in (in however long that may be) we could have problems if those lines are not more defined.
🙂 Smiley face to show I’m smiling
LikeLike
Certainly as this blog and its comments stand – it’s quite interesting to see that the two people who have commented at me and made assumptions about what I have said, have remained anonymous – would those comments been written in the same way if they were open? It would be nice to see who I am talking to for the purpose of this debate 🙂 This is a friendly suggestion btw
LikeLike
Andy, this thread is the first appearance of “IrritatedResearcher” on this blog. I hope she will come back regularly as he has made interesting and valid points. The fact I have no idea who they are makes strictly no difference. Nanonymous however is a regular contributor to discussions here. Nanonymous has offered very insightful scientific comments on the stripy controversy and I am thankful for that. Again, it makes no difference to me who is Nanonymous.
At that point in the discussion, I suppose I should confess that although I defend the right, benefit and legitimacy of anonymous commenting, I have, once, identified and revealed the identity of an anonymous commenter on my blog. This was a somewhat extreme case (and that person later apologised for this ) but I am still not 100% sure that was the right decision.
LikeLike
That’s fine, totally understand – but what you do have to understand – that as this way of communicating opens up in research – some will feel uncomfortable by not knowing who they are speaking with (this is what I’m trying to highlight) as everyone is different – some use social media anonymously, some open, some not at all – everyone has their own reasons.
As for revealing the identity of an anonymous commenter – I think that should come down to that person. If anything is ever said untoward, unprofessional or even derogatory then it is down to the moderator to either kick that person off the site or contact them directly to address the issue (hence the usefulness of moderated comments) as naming and shaming is not a helpful approach should that ever be the case.
Either way, we have to stay clear of the notion that everyone sues the Web in the same way and all too often people can be driven away from its use by the actions of others.
LikeLike
@ Andy
An apology: I did mistakenly believe you were associated with F1000. My mistake.
As for “assumptions about what you said”, your article, while maintaining some balance, packed in a huge number of negative memes regarding anonymous commenting. There are several passages along these lines:
“Post-publication review also has the potential for bias via preconceived judgements. One researcher may leave harsh comments on another’s research based on the fact they do not like that person: rivalry in academia is not uncommon. Trolling on the web has become a serious problem in recent times…”
The reader is obviously being guided to believing that scientists are trolling each other via anonymous comments. It’s a plausible idea, but the point I was trying to make is that it’s not borne out in practice. If, instead of analysing the medium, you had actually examined the message (i.e., read and understood the comments), you would have found that the great majority make very specific factual and scientific points. Sometimes they are very damaging, but that doesn’t mean they are incorrect, trolling or without value.
Nominative comments, such as on PubMed Commons, are a great way for established researchers to give authoritive feedback. But when less secure researchers wish to make a serious criticism or point to published data indicating misconduct, anonymous commenting is I believe necessary. It is certainly the case that PubPeer, Retraction Watch and Paul Brookes’ ex-site have brought to light a huge amount of misconduct that existing systems appear unable to detect or police. So I see a real need for both types of commenting.
LikeLike