Monday, July 25, 2016

What quality controls are utilised by PLOS ONE when selecting reviewers? Who is deemed eligible?

In 2011, I expressed concern about the PLOS ONE business model and its associated review process. My worries were focused on the use of what some have called light or “lite” peer review, and the “pay-to-publish” system used by PLOS ONE (and now by many other publishers). My worries were subsequently recorded on the PLOS ONE Wikipedia page

Recent personal experience has increased my concern, and left me wondering about the way in which reviewers are recruited by PLOS ONE. 

On 12th July, I received an email from a PLOS ONE academic editor inviting me to peer review a paper. I won’t say what the paper was entitled, or who the authors were, but it was on the topic of open access journals.

Since I am a blogger/journalist rather than an academic I was surprised to receive the invitation, and emailed PLOS ONE with the following question: “I have had an invitation to review the above paper. Can you point me to the rules on the eligibility of PLOS ONE reviewers?”

I received the following (I assume boilerplate) reply:



This did not address my question, so I also emailed the academic editor whose name had been at the bottom of the invitation. I am not going to name him, but I will say that he is based at Universidad de Las Palmas de Gran Canaria, “Was it you who sent me an invitation to review the above paper?” I asked. “If so, I am wondering why you invited me. Can you say?”

I received the following response:


Grunt work

Let me be quite clear at the outset: I had and have no interest whatsoever in reviewing this or any other scholarly work, not least because there is absolutely no incentive for me to devote my time to reviewing papers. Moreover, the one time I did agree to review anything for an academic journal (an editorial rather than a paper), my suggestions were all rejected on the grounds that “the author says he is too busy to make the changes you suggested.” Clearly I had not made very good use of my time!

But as I say, my first response on receiving the PLOS ONE invitation was to wonder whether it is inappropriate for non-academics to review scholarly papers.

With these thoughts in mind I tweeted the invitation under the strapline “PLOS ONE invites journalist to review scholarly paper”. Somewhat to my surprise, everyone who responded said that they saw no problem with my reviewing a scholarly paper on open access (although it could not presumably be defined as “peer” review). Their reasoning was that they are confident that I have the necessary expertise. And Roger Schonfeld commented, “I’d like to see expertise welcomed into the scholarly conversation without regard to academic affiliation.”

On reading these responses I recalled that some OA advocates maintain that an important benefit of OA is that it encourages members of the public to take a greater interest in science, and to even take part in the process themselves – by means of “citizen science”.

A few days later I read this in the Times Higher: “the open access movement in academic publishing, while far from perfect, means that more research papers than ever are available to those outside universities. Are we not entitled to expect graduates in the workforce and the general public to show the curiosity and commitment to engage with research findings relevant to them?

It added: “Academics should not hide in the ivory tower, but neither should non-academics refuse to re-enter it after they graduate. Its doors remain wide open. Surely it is not too much to ask for policymakers, journalists, non-governmental organisations and industry researchers to make a greater effort to access its treasures?”

That all sounds laudable enough, but I wonder if the authors of the Times Higher article (two cultural studies researchers) would want their scholarly papers reviewed by members of the public.

As it happens, I tend these days to turn a somewhat sceptical ear to people when they start extolling the virtues of citizen science. As I put it in an interview last year, when researchers talk about the benefits of citizen science they usually have in mind a scenario in which the public’s role is simply to provide free labour for crowdsourcing projects. Citizens are despatched to count butterflies, bugs or birds, stare at images of galaxies on their computer, or “swab and send”, and then feed the data back to bona fide scientists in the lab.

“This activity is clearly valuable,” I suggested, “but seems to imply that the role of citizens in science is to do the grunt work, while the real science continues to be undertaken by professionals.”

Left wondering

That said, I was still intrigued as to how and why I had been invited to review a paper for PLOS ONE. So I persisted in my attempts to get an explanation, putting to PLOS ONE the following three questions:

·       Does PLOS ONE allow or not allow people to review a paper where they are not a member of a university or other research institution?

·       Does PLOS ONE have a published list of the criteria by which reviewers can be deemed to be eligible to act as a reviewer? If not, can you tell me what they are anyway?

·       On what grounds was I invited to review the paper in question, and why was it later decided that the invitation was a mistake?

It was not easy to get an explanation out of PLOS ONE, despite two tweets from the publisher saying that it was looking into the matter (here and here).

Eventually (on 22nd July) I received the following email:

Academic Editors at PLOS ONE have great autonomy when selecting reviewers, and the decision as to whether an individual expert is eligible to act as a reviewer for a specific manuscript lies with the Academic Editor. We have been in touch with [the academic editor] and he has noted, as we mentioned, that the reasons why specific individuals are approached as reviewers are numerous and may vary depending on the submission. In relation to the invitation you received [the academic editor] has reiterated that this was issued in error, he apologized to you for this error when you first raised this to his attention.

From this I conclude that there is in fact no policy on reviewer eligibility at PLOS ONE, although it would have been nice if the publisher had directly acknowledged this.

So I am left wondering how PLOS ONE academic editors make their decisions when recruiting reviews. Presumably there will be quite a diversity of views on eligibility. Is such a loose arrangement really satisfactory?

Interestingly, a few days later Chealsye Bowley, a librarian at Florida Gulf Coast University tweeted about a related problem she had experienced with F1000Research. She had been informed that as she does not have a PhD she is not eligible to submit a paper to the publishing platform.

The good news is that F1000Research does have a published authorship policy. The better news is that the company responded to Chealsye’s concerns by proposing to amend the policy to cater for people like her (although some still view the proposed amendment as unsatisfactory).

Like Chealsye I too do not have a PhD. I did spend a year at Moscow State University in 1981 studying the “village prose” writers (e.g. Vasily Belov and Valentin Rasputin), but the interminable queues I had to stand in at the Lenin Library simply to request access to the books I needed – along with the constantly flowing Столичная vodka – proved too great a distraction. I never finished my thesis.

Not very encouraging

However, the fact that I do not have a PhD would not have been known by the PLOS ONE academic editor since I have no CV online. Presumably, therefore, he withdrew the invitation when he realised I was not a researcher. But should he not have established that before sending me the invitation? When I later asked him how the mistake had occurred, and how I had been targeted, he did not reply.

But there are some clues as to how I was selected, and I don’t find them very encouraging. Let me explain: 

The invitation I received (below) was sent to me at 12:08 on 12 July 2016.


When I looked at my blog statistics I noted that four minutes earlier someone from the University of Las Palmas De Gran Canaria had accessed one of the pages on my web site (this one).


This was the only page the visitor from Gran Canaria accessed that day, so I am inferring that the vetting process went something like this:

1.     The academic editor does a Google search on “open access” and sees my name in one of the hits.

2.     He clicks on the link in that list and is sent to a page on my site.

3.     He quickly scrolls down that page, copies the email address at the bottom, and sends off his invitation.

This suggests that the academic editor did not think to check whether I was a researcher. If so, it is a pretty telling mistake. After all, the banner on the top of the page he accessed on my site quite clearly states that I am an “independent journalist”. In addition, the email addresses he copied clearly describes my status – richard.poynder@journalist.co.uk.

If that’s right, then the mistake the academic editor made was to spend far too little time establishing the eligibility of the person he invited to review his open access paper – around four minutes in fact. This is of course speculation on my part as the academic editor stopped responding to my emails.

This leaves me with three further questions:

1.     Do PLOS ONE editors routinely only do four minutes research when looking for, and establishing the eligibility of, a reviewer, or was this editor in an uncharacteristic hurry?

2.     Does PLOS ONE have no quality controls in place to enable it to itself verify the appropriateness of reviewers being recruited in its name? Does it simply leave everything to the academic editor?

3.     If I had not questioned the invitation, but just sent in a review, any old review, and that review had met the needs of the academic editor, would anyone have known that a “mistake” had been made? After all, PLOS ONE does not operate open peer review – its guidelines state: “Reviewers are anonymous by default. Reviewers’ identities are not revealed to authors or to other reviewers unless reviewers specifically request to be identified by signing their names at the end of their comments.”


Serious issues

And if the situation at PLOS ONE is as I infer it to be then I think it raises some serious issues. As Danny Kingsley has pointed out, problems with peer reviewers are now a big, and growing, problem. “In August 2015 Springer was forced to retract 64 articles from 10 journals, ‘after editorial checks spotted fake email addresses, and subsequent internal investigations uncovered fabricated peer review reports’”, she wrote recently. “They concluded the peer review process had been ‘compromised’.”

Kingsley added, “In November 2014, BioMed Central uncovered a scam where they were forced to retract close to 50 papers because of fake peer review issues. This prompted BioMed Central to produce the blog ‘Who reviews the reviewers?’ and Nature writing a story on Publishing: the peer review scam.”

Is not PLOS ONE laying itself open to similar scrapes and scandals? (As it is, PLOS ONE has already featured on Retraction Watch a number of times).

It is also worth noting that in the Nature article that Kingsley cites (published in 2014) the authors point out that the editorial system used by PLOS ONE (Editorial Manager) is insecure, since it “actually sends out a password, without prompting, whenever it asks a user to sign in, for example to review a new manuscript.” Two years later this is still happening, as I can now attest.

All in all, I feel my experience suggests there are two, related, issues here. First, there appears to be inadequate checking of reviewers at PLOS ONE, and apparently no official policy on reviewer eligibility.

Second, in describing my invitation as a mistake, PLOS ONE would seem to assume that the involvement of citizens in science should, at best, be no more than as occasional passive readers of papers, and/or fodder for citizen science grunt work.

As I said, I have absolutely no wish to review papers myself, but I am confident that many citizens would be more than willing to contribute. And so long as they had the necessary expertise they would probably do a better job than many academics (I say that having read quite a few reviews in my time). Either way, I would suggest that establishing the expertise of a reviewer (whoever they might be) requires more than a quick Google search.

Moreover, if, as many maintain, the boundary between citizens and scientists should be more porous, is PLOS ONE sending out the right message to the public with incidents like this? I also suspect that had I been a researcher I would have received a more detailed and respectful response to my enquiries from PLOS ONE.

The Ivory Tower is dead, long live the Ivory Tower!

15 comments:

Anonymous said...

"...I also suspect that had I been a researcher I would have received a more detailed and respectful response to my enquiries from PLOS ONE..."

As a researcher, I find this assumption most amusing!

pippasmart said...

I can understand your concern, but as a journal editor myself (Learned Publishing), I frequently (almost always?) use non-academics to review articles due to the nature of the content. Maybe they do have a PhD, but that is irrelevant to me - I choose them because they have knowledge in the area that the author has written on and can therefore judge its validity. Sometimes I select people specifically because they are not experts but I want their opinion as a general reader ("is this interesting/sufficiently informative" etc.).

So, in the case you cite, my assumption is that PLOS had an article about a subject on which you have written a great deal and can therefore be expected to know a great deal about. I think your suggestion of how you were selected was a little disingenuous - I suggest that the editor had looked on other sites or spoken to people, discovered your name as an "expert", then visited your site to read some of what you had written to confirm your knowledge, and then contacted you.

Richard Poynder said...

Thank you for your comment Pippa. You too are speculating here of course. All I can do is once again invite the academic editor to explain the process he went through before inviting me to review the paper, and why he then withdrew the invitation when I asked him why he had asked me.

daniele said...

Dear Richard, thanks a lot for your message.

Disclaimer: I am an Academic Editor at PLOS One. Academic Editors are academics who agree to do some not remunerated work for the journal.
The reviewers that we choose to assist our decisions are also supposed to check whether they have the time, they feel qualified, they have no conflicts of interests. Then, they submit a recommendation. Reviewers also are not remunerated.
The ultimate choice is of the Academic Editor. It's certainly a good thing to be assisted by reviewers, but it's not mandatory. In the great majority of the cases, several reviewers are invited, and usually editors are satisfied when they got at least two reviewers who accept to help them.

The academic editors at PLOS One are thousands, and the journal publishes around 30000 papers per year.

Sorry for this lengthy disclaimer, but I think that on a public platform is better to state some things first. It should be clear that here "speaking as academic editor of PLOS One" does not mean that I speak on behalf of the journal, or that I have any interest beyond contributing to diffusing good science. So, I speak as myself, from my experience as Editor as PLOS One (and in general).

Now, thanks again for pointing out a problem. Problems must be spotted and solved.
In this case, your doubt is certainly legitimate, and the Editor should have answered your query with something more extensive. We don't know what the mistake is, a spelling error, a copy-paste error, or as you suggest, an error in considering you suitable or interested when you have doubts about it.

The thing is, as you say, that there is no clear rule for editors to invite reviewers. They can rely on their feelings, in the light of receiving the most helpful comments. Then it's their job to see if these comments are useful or not.

As an Editor I have received many excellent comments (being then very happy of my choice of the reviewers), and some really poor ones (for which I said to myself "too bad, your choice was not the good one clearly").
I also (a couple of times) have invited non academics. The number is too low to generalize, but their comments were great, with a fresh perspective.

Regarding your case I think that there was a problem of communication. I am interested in whether the editor really thought that you could help, on some grounds, and then had an awkward reaction to your question, or if he made a trivial mistake and did not mean to invite you. PLOS One also uses an automated system to suggest reviewers, based on keyword match I think. It's mostly useful, but an editor should always check the suitability of the suggested name.
Anyway none of this can be inferred from the story.

The thing I am not ok with is the generalization. I was pointed to your article by a twitter user who titled "The way in which reviewers are recruited at PLOS One". Not your responsibility of course.
But I think that you shift the attention from a single, actual problem, when you put in the same post worries about pay-per-publish. This system is good for open access, but has some problems, as those pointed out by Brembs here http://bjoern.brembs.net/2016/04/how-gold-open-access-may-make-things-worse/

but I really don't think it's connected to the quality of the papers.

Also the presence on Retraction Watch of PLOS One papers is anecdotal, given the huge number of published papers. The number of retractions is indeed the only thing that clearly correlates with Impact Factor, being led by glamour (and overly respected) journals.

http://journal.frontiersin.org/article/10.3389/fnhum.2013.00291/full

Most importantly, Open Access itself is not correlated with bad or sloppy reviews (nor is PLOS One to be associated with Open Access as a whole).

Anyway, thanks again for pointing to the communication problem that you had with one of the 6000 editors.
Sorry for the lengthy comment, I hope it's useful.

All the best, and keep up the good work.

Daniele Marinazzo

Stevan Harnad said...

Generic ”Megajournals”?

My 2p worth:

(1) You, Richard, are an appropriate reviewer for some papers on open access or on publishing. (PhDs and academic status have nothing to do with it.) I would have chosen you for some papers, if I were editor.

(2) PloS ONE is not a journal: Nothing that has thousands of editors and publishes 30,000 journals yearly is a journal.

(3) You imagine that peer review is a more solemn and systematic a process than it actually is: There is huge variation in its rigour from journal to journal and discipline to discipline.

(4) But at least with a journal, with a track-record, what you see is what you are likely to get.

(5) With PloS ONE you don’t just get “peer review lite” (whatever that means): You get a pine-nut in a poke. [I abjure the usual sociopathic cliché.] Impossibl

Stevan Harnad said...

Correction above:

Nothing that has thousands of editors and publishes 30,000 papers yearly is a journal.

Stevan Harnad said...

Correction #2 above:

(5) Impossible to know what you're getting.

Mike Taylor said...

Seems to me like a classic case of "Move along, nothing to see". Here is what I see as the likely scenario:
1. Editor invited to handle a paper about open access.
2. Invites a recognised expert on the subject to review.
3. Gets a slightly snotty message asking about it.
4. Thinks "I can do without this, I'll find someone else" and withdraws the invitation.
I imagine I would have done the same in his position.

For the record: I firmly believe that reviewers can and should be drawn from the entire population of those with relevant expertise, completely irrespective of their formal qualifications or affiliations. And F1000's "must have a Ph.D to be an author" policy is a horrifyingly elitist blunt instrument, which leaves me feeling absolutely no inclination ever to send my work to them.

Richard Poynder said...

Thanks for your comment Mike. Your scenario is more speculation of course, which reminds us that one thing the open access movement has generally failed to do is to rid us of the secrecy, and the unaccountable gatekeepers, that have for so long plagued scholarly communication. At least F1000 is trying to increase transparency of process.

Klaas van Dijk said...

My experiences until now with communicating with members of the staff of publisher PLOS show similarities with the experiences of Richard Poynder. I have received several so-called boilerplate responses (or no response at all) and I have received loads and loads of auto-replies.

All of these auto-replies have a different case ID and all of them state something like "We will respond to your email as soon as we are able". This simply does not happen.

See https://scholarlyoa.com/2016/01/14/another-controversial-paper-from-fronters/#comment-405567 for some details of my experiences with Iratxe Puebla (currently the Managing Editor at PLOS ONE).

PLOS has an extensive policy for authors, reviewers, editors and readers for all journals to declare competing interests, see http://journals.plos.org/plosone/s/competing-interests

Members of the staff of the journal PLOS ONE do not need to disclose such details. Iratxe Puebla is for example acting as a consultant for COPE, but this information is not listed at http://journals.plos.org/plosone/s/staff-editors

Karen Shashok said...

People with no academic or institutional affiliation can have knowledge, experience and expertise their age- and discipline-matched counterparts in the Ivory Tower may lack. It’s healthy for reviewers and editors to seek expertise outside of academia if they think someone is likely to have useful feedback to offer.

I think this was the situation since Open and Shut is awesome and very well known among those who follow developments in academic publishing and open access. You were given a golden opportunity to help some researchers, somewhere, improve their article. You underestimated your skills, or chose to focus on a non-issue. Nice mountain! Too bad it’s built out of a molehill.

Your PLoS ONE academic editor did not respond to you in a way you expected, and your reaction is understandable – but is not a reaction that other non-academic invited reviewers would share, I think. When I’ve been invited to review for a journal I’ve always felt honored and delighted to help. (My highest academic degree is a BA and I’ve been self-employed all my professional life. That hasn’t been an issue so far for editors who’ve contacted me.) If I see from the invitation and abstract that I don’t have the necessary expertise to contribute useful feedback, I decline the invitation, and that’s the end of it. Takes maybe 5 minutes.

The PLoS ONE system of editorial quality control is decentralized, and there are no detailed SOPs for their thousands of experts who provide unremunerated editorial support. I think PLoS is aware that the lack of transparent information about their processes worries people, and are taking steps to provide more information. Meanwhile there will be considerable variation in how individual academic editors and reviewers handle their appointed tasks.

But even at journals with fine-grained review procedures (some but not so many) that are fully reported on their website (not many at all), human errors and oversights occur – ask anyone who publishes in journals or peer reviews for them. This is nothing to do with the PLoS model or with open access.

(BTW, what need was there to identify the institution you surmised the academic editor is from? Are you suggesting that it has anything to do with this person’s competence or behavior? I found this off-putting, but maybe that’s just another molehill.)

Richard Poynder said...

Dear Karen,

Thanks for taking the time to post a comment. Just to clarify:

1. I have no desire or wish to review any scholarly paper, be it published by PLOS ONE or any other scholarly publisher. As such, there was no golden opportunity here that I passed up.

2. You say that when you are invited to review papers you take a view on whether or not you are qualified to review them. In my case this was not possible as the abstract was not sufficient for me to make that decision (had I been interested in reviewing the paper), and when I logged into Editorial Manager to form a better view of what the paper was about the system did not let me proceed unless I filled in a box saying what institution I am a member of. This was problematic as I am not a member of an institution.

3. I don’t believe that the points I made are a non-issue. As you yourself acknowledge, there are at the very least problems with transparency in PLOS ONE’s processes. You say that PLOS ONE is aware of this and is taking steps to provide more information. Can you clarify this? Are you speaking for or about PLOS ONE?

4. I have no problem in principle with decentralised systems, but decentralisation without transparency appears to be problematic.

5. You say that I surmised who the academic editor is. I did not surmise this, his name was on the invitation I received, and PLOS ONE confirmed the name of the institution he works for.

6. There was no intention whatsoever to suggest that the institution concerned has any bearing on the academic editor’s competence or behaviour, nor do I believe it does. I identified the university simply in order to support my interpretation of how and why I was invited, which if correct is worrying.

7. You say that even journals with fine-grained review procedures experience human errors. My point in the post was not about human errors but the nature of the PLOS ONE system. I am concerned about its non-transparent processes; the huge number of papers it publishes (and the consequent high degree of automated processes this requires); the way the business model encourages a high number of papers to be submitted to it; and an unwillingness to engage with people who experience problems and/ or have concerns with PLOS ONE.

8. If it is true that I have made a mountain out of a molehill, why have the academic editor and PLOS ONE consistently declined to explain why I was invited and then told me that my invitation was a mistake? Why would they not want to explain it, if only privately?

Richard Poynder said...

See also this post: https://scholarlyoa.com/2016/08/23/ongoing-questions-about-plos-ones-peer-review/

Unknown said...

Your post raises some valid points, but as someone who has been involved in academic publishing in many roles (author, reviewer, editorial board member, "academic editor", editor-in-chief) for both OA and subscription journals, there are a few remarks I'd like to make.

1/ I do not know of ANY journal that has a stated policy about who can be a reviewer or not I'm not saying they don' exist, but I'm sure it is something quite rare).

2/ Some of the best reviews I received as an editor came from graduate students without a PhD. Some of the worst reviews I got came from full professors at reputable universities, who just should have said "no" because they clearly lacked the necessary time to do a thorough job and thus missed the most obvious fatal flaws.

3/ I know a lot of idiots with a PhD. I know a lot of qualified people that don't have one. Although in most cases an editor will invite a reviewer who does have a PhD, this is certainly not a rule written in stone and, as far as I am concerned, totally irrelevant. What counts is whether the reviewer has the necessary expertise.

4/ Every editor makes mistakes. If they do this often, they're simply bad editors and should be removed. But even the best editors are only human.

5/ You underestimate yourself. You're a well-known specialist on open access. I would not accept to edit an article on OA myself as I do not think I'm qualified for that, but if ever I did, I would not see any problem with inviting a recognized specialist, even if he has no PhD or formal academic affiliation. I am sorry that you let one unfortunate experience make you unwilling to even consider reviewing a paper.

6/ PLOS ONE is by now the largest journal around, publishing a significant proportion of all papers indexed in, for example, MEDLINE. Given the sheer number of articles published, I think the number of retractions is actually below the mean of most journals. They have over 5000 academic editors and, again, given that huge number some will be sub-par. As far as I know, really weak editors get weeded out, but that means of course that by the time this happens, they have made some mistakes. However, I don't really see a way to do this better...

Unknown said...

I agree that the editor's response is difficult to understand. I don't see anything wrong with asking a non-academic to review a paper in principle -- though I haven't found reason to do that yet. If the person doesn't respond in a given time, then the answer is "no". If the person complains, say "I'm sorry for troubling you" (if nothing else) and move on.

I would like to add a little to Wim Crusio's point 3 (above).

I think following comment somewhat misses the point of scholarly publishing

"“This activity is clearly valuable,” I suggested, “but seems to imply that the role of citizens in science is to do the grunt work, while the real science continues to be undertaken by professionals.”"

This probably is true in practice. Nevertheless, good science and good scholarship are not (or at least should not be) limited to academics. We city slickers may say "dumb farmer"; however, a successful farmer has to be pretty smart, in fact.

We (as academics) are better trained on how to communicate in such a medium (compared to citizen science), but quality scholarship can (and should) be recognized wherever it comes from. To carry the farmer example further, the chance to stumble an interesting topic to write on, the challenge of obtaining the research materials (specific types of crops perhaps, and then to navigate through pay walls and limited library support), the difficulty of understanding such specialized (and often opaque) papers, and the challenge of writing an original paper of one's own in a style compatible with an academic publication are all things that would make it difficult for a farmer in Indiana on his own dime to produce.

We should not be prejudice toward a work simply because it was written by a farmer (to carry this example).

Moreover, when I have reviewed very poorly written papers by some academics, I often comment to those authors to consider the janitor in some obscure part of the world picking up this paper out of a waste can. What would s/he think? Consider the student who is wandering through the stacks one day and just happens to stumble on this paper when opening a random booklet. (I guess now, it would be surfing the web, but I think the point is clear.) We don't know who is going to read our work. What would that student think?

Papers should not be written with the mind that no one other than some other very closely allied specialist is going to read that work. Indeed, this is the most exclusionary of all writing. Newspapers have to make some attempt to connect their readers. Likewise, journal publications should make at least a feeble attempt to provide a context and connect with the reader. As Strunk & White wrote, "we should consider that the reader needs help" (and probably lots of it). As far as we have the ability, our writing should be for the readers, and we should never see these publications as little more than trophies to be stacked up for use in gaining further academic promotions.

Consider what Benjamin Franklin wrote:

"If my hypothesis is not the truth, it is at least naked;
for I have not with some of our learned moderns
disguis'd my nonsense in Greek,
cloth'd it in algebra or adorned it with fluxions."

Franklin, by the way, was not particularly an traditional "academic".