Articles

Peer Review and the Internet

Peer-review has been the cornerstone of quality control in academia, including science and medicine, for the past century. The process is slow and laborious, but a necessary filter in order to maintain a certain standard within the literature. Yet more and more scholars are recognizing the speed, immediacy, and openness of the internet as a tool for exchanging ideas and information, and this is causing some to question the methods of peer review. A recent New York Times article discusses this issue.

This issue is very relevant to Science-Based Medicine as this is in part an experiment – an attempt to produce a high quality, editorially filtered, but not peer-reviewed, online journal. Our process here is simple. Outside submissions are reviewed by two or more editors and typically are either accepted with minor revisions or rejected. In addition we have a staff of regular contributors – those who have a proven track record of producing high quality articles. There is no pre-publication review for their submissions, and they are able to post directly to SBM.

Because many of the issues we cover are timely, we emphasize speed of publication. Therefore copy-editing is done post-publication – the notion being that our readers can tolerate a few typos in order to gain access to material more quickly.

In addition our pieces are, essentially, crowd sourced. SBM editors and readers provide feedback in the comments, often pointing out ambiguous wording or even outright errors, which can then be quickly fixed or clarified. Editors also provide direct feedback to authors if a more serious issue emerges (which is rare, but happens) and steps are taken to transparently fix such issues. This is, in a way, post-publication peer-review.

By contrast, traditional peer-review is designed to take a long time to produce the highest quality article prior to going to print (even if “print” is online). The strength of this process is that several editors and peer-reviewers have thoroughly gone over the submission, corrected errors, fixed ambiguities, added missing insights, reviewed methodology, checked references, and made sure that the author’s conclusions do not overstep the data. At least this is the ideal – we often write about peer-reviewed publications on SBM that fall far short of this ideal. In the end, peer-review is only as good as the editors and reviewers. I have both been a writer and reviewer, and the process does have many strengths. In the end the article that gets printed is much improved over the original submission.

The weaknesses of the process, however, are the long delay to publications – months or even years. Further, the process is a bit hit or miss and depends largely on the quality of the reviewers. Individuals are quirky, and they may have biases or missing information that will hamper the quality of their review. The more people there are involved in the process the better the quality is likely to be.

As academia grapples with the internet age, the challenge is to rethink the process of peer-review. Specifically, are there ways to leverage the power of the internet to make the review process faster and better? This has happened in the technical sense – the traditional process of peer review is now often done online. For the last article I reviewed I was not sent a hard copy or even an electronic copy. I signed onto a secure website where I had access to the article, and I submitted my review entirely online. This was convenient and also a time-saver. But still the process was slowed by the need to choose reviewers, wait for them to accept and then for those specific reviewers to review the paper. And in the end opinions were solicited from only two reviewers.

Imagine an alternate process by which an article is published online, either on an open site or a secure site that only experts have access to. Then dozens or hundreds of experts can comment on the paper, providing feedback directly to the authors in addition to the editors, who can also respond to the commenters. The result would be more of a dynamic conversation than you get with the current review process. But most importantly, in my opinion, is that you would get a broader range of opinions, and a far greater chance to detect error or bias. An editor or editors can oversee the process, and once it has played itself out the final version of the paper can be published to the public, and become part of the official literature.

Now is the time for experimentation, in my opinion. I don’t know yet what the optimal system would be, but I think a hybrid of peer-review and open publication will emerge. There still needs to be some filter, and some editorial process selecting articles, overseeing feedback and revisions, and approving the final version. Feedback from many experts should be solicited in some fashion, and the process should be interactive.

There are many details to experiment with, however. Should commenters be named or anonymous? Anonymous feedback may be more candid, but people should be held accountable for their expert opinions as well. How should commenters be limited, if at all? Should there be a period of feedback open to the public, or should pre-publication access only be granted to experts, and how should that be limited? (How narrow a field of experts should apply?)

There is also the issue of presenting new research in an open forum prior to adequate review. Will the press troll these pre-review sites looking for juicy stories based upon flawed research that would never ordinarily make it to publication?

The strengths and weaknesses of any new process of peer-review will not be fully appreciated until it is implemented, which is why we are in a period of experimentation. But the experiment is worth doing. In the end we may gain a better, faster and more transparent review process.

Posted in: Medical Academia

Leave a Comment (15) ↓

15 thoughts on “Peer Review and the Internet

  1. windriven says:

    Wiki-science with adult supervision.

  2. cervantes says:

    Well yeah but you don’t post original research here, you just talk about stuff. It’s not remotely comparable.

  3. Scott says:

    There is also the issue of presenting new research in an open forum prior to adequate review. Will the press troll these pre-review sites looking for juicy stories based upon flawed research that would never ordinarily make it to publication?

    To give a (scary) example on a subject SBM often covers, consider articles by Wakefield or the Geiers. Bad enough when such show up in Medical Hypotheses; if they appear next to REAL research they’ll gain additional credibility.

  4. art malernee dvm says:

    A healthcare giver will be able to share this “high quality, editorially filtered, but not peer-reviewed, online journal” with patients on a ipad like device in the exam room,bedside, or on video conference. The doctor and patient or in my case the client will not be paying a fee to use it. The journal will not even be under corporate or government influence. Sound to good to be true. I hope a SBM journal can figure out a way to stay free.

  5. B Hitt says:

    I think it would be great if the online versions of peer-reviewed journals had comments sections like this one to facilitate post-publication review, kind of like the Q & A after research talks at conferences but the discussions could get more in-depth.

    Another thing that the internet has done has been to make it less important which journal you publish your results in. While in the past, your paper was likely to be read by far more people if you published in a high-profile journal, now everybody gets their literature online. Your paper is just as likely to come up in a PubMed search regardless of which journal it’s in, leveling the playing field to an extent.

  6. daedalus2u says:

    B Hitt, many online journals do have space for comments. PLoS does, BMJ does.

  7. WilliamLawrenceUtridge says:

    Will the press troll these pre-review sites looking for juicy stories based upon flawed research that would never ordinarily make it to publication?

    Yes.

    Citizendium, wikipeidia’s more restrictive, less popular boring cousin, allows this to a certain extent and requires open editing using a real name. A problem I’ve noticed is cranks that get booted from wikipedia tend to end up there instead. Since CZ allows limited speculation from ostensibly informed editors, they can run a bit further with their articles and ultimately I find the articles I work on on wikipedia read better and are more comprehensive. See the satanic ritual abuse article on both for instance. Ultimately it always comes down to the reviewer, whoever has their eye on the page. Richard Smith’s book The Trouble with Medical Journals covered this topic in the book – an interesting read.

    @Scott

    The problem with the authors you are citing is that they deliberately, either through willful stupidity, active malice or thoughtless bullheadedness, sought to use the existing systems, predicated on honesty, to promote a dishonest agenda. For the most part, this kind of research works. But when money (or pride) get involved, you run into problems and dishonesty, and it takes a while before the relevant community can muster the resources to review claims in detail and state unequivocally that the claims are nonsense. The cycle has happened with vaccination, creationism, satanic ritual abuse, N-rays, homeopathy’s serial dilution, and is now happening with acupuncture. It’s like all trust – human society runs on it, but the default is often to trust people to be honest. It takes time and effort to verify that trust has been broken. But when it happens, you can never really get it back.

  8. Scott says:

    @ William:

    My point is that we should consider just how much MORE damage they could have done in an open-publication model.

  9. Maz says:

    @ cervantes,

    The post here do not contain original research, but you could say the same about many journals that focus on review articles. In both cases, the authors are analyzing existing research or claims and analyzing their plausibility in the wider context of scientific knowledge.

  10. Kausik Datta says:

    But in an open review system like the one proposed in the NY Times article, how does one separate the proverbial grain from the chaff? Would the authors of a study be obligated to provide equal weight to reviews from qualified professionals as well as unqualified hacks? For example, what if Dana Ullman and Sid Offit claim expertise and clamor for access to the so-called expert portal for commmenting on studies? Who would arbitrate and adjudicate in such situations? Is this going to be a process of “publication by press”? I am afraid, very afraid.

  11. Necandum says:

    @Kausik

    Publishers already select reviewers through one process or another, so I don’t seen why an open review system has to be any different. Besides, it could become a self-sustaining system, if, for example, one received access to the ‘expert portal’ after having several relevant articles published.

  12. “Will the press troll these pre-review sites looking for juicy stories based upon flawed research that would never ordinarily make it to publication?”

    Almost certainly not. For one thing, that would require lots of time AND abstracts that highlight potential juiciness. The problem with the current system is the press-releases that journalists do use to decide what articles to write about. And that factor will likely get worse as mainstream science journalism is increasingly cut back. The desire to sensationalize won’t get better either.

    The only shred of hope out there is that the mainstream press will use blogs to troll the interwebs for reputable, and reasonably concluded, current research.

  13. TILL STANDARDS, SUSTAINABILITY AND SCALABILITY ARE TESTED:
    PEER COMMENTARY IS SUPPLEMENT, NOT SUBSTITUTE, FOR PEER REVIEW

    See: Peer Review Reform: bit.ly/peer-review-reform

    Harnad, S. (1978) Inaugural Editorial. Behavioral and Brain Sciences 1(1).
    http://www.ecs.soton.ac.uk/~harnad/Temp/Kata/bbs.editorial.html

    Harnad, S. (ed.) (1982) Peer commentary on peer review: A case study in scientific quality control, New York: Cambridge University Press.

    Harnad, Stevan (1985) Rational disagreement in peer review. Science, Technology and Human Values, 10 p.55-62. cogprints.org/2128/

    Harnad, S. (1990) Scholarly Skywriting and the Prepublication Continuum of Scientific Inquiry Psychological Science 1: 342 – 343 (reprinted in Current Contents 45: 9-13, November 11 1991). cogprints.org/1581/

    Harnad, S. (1996) Implementing Peer Review on the Net: Scientific Quality Control in Scholarly Electronic Journals. In: Peek, R. & Newby, G. (Eds.) Scholarly Publishing: The Electronic Frontier. Cambridge MA: MIT Press. Pp 103-118. cogprints.org/1692/

    Harnad, S. (1997) Learned Inquiry and the Net: The Role of Peer Review, Peer Commentary and Copyright. Learned Publishing 11(4) 283-292. Short version appeared in 1997 in Antiquity 71: 1042-1048. Excerpts also appeared in the University of Toronto Bulletin: 51(6) P. 12. cogprints.org/1694/

    Harnad, S. (1998/2000/2004) The invisible hand of peer review. Nature [online] (5 Nov. 1998), Exploit Interactive 5 (2000): and in Shatz, B. (2004) (ed.) Peer Review: A Critical Inquiry. Rowland & Littlefield. Pp. 235-242. cogprints.org/1646/

    Harnad S. (2002) BBS Valedictory Editorial (2002) Behavioral and Brain Sciences 24 users.ecs.soton.ac.uk/harnad/Temp/bbs.valedict.html

    Harnad, S. (2003) PostGutenberg Peer Review the invariant essentials and the newfound efficiencies users.ecs.soton.ac.uk/harnad/Temp/peerev.pdf

    Harnad, S. (2009) The PostGutenberg Open Access Journal. In: Cope, B. & Phillips, A (Eds.) The Future of the Academic Journal. Chandos. eprints.ecs.soton.ac.uk/15617/

  14. TILL STANDARDS, SUSTAINABILITY AND SCALABILITY ARE TESTED:
    PEER COMMENTARY IS SUPPLEMENT, NOT SUBSTITUTE, FOR PEER REVIEW

    See: Peer Review Reform: http://bit.ly/peer-review-reform

    Harnad, S. (1978) Inaugural Editorial. Behavioral and Brain Sciences 1(1).
    http://www.ecs.soton.ac.uk/~harnad/Temp/Kata/bbs.editorial.html

    Harnad, S. (ed.) (1982) Peer commentary on peer review: A case study in scientific quality control, New York: Cambridge University Press.

    Harnad, Stevan (1985) Rational disagreement in peer review. Science, Technology and Human Values, 10 p.55-62. http://cogprints.org/2128/

    Harnad, S. (1990) Scholarly Skywriting and the Prepublication Continuum of Scientific Inquiry Psychological Science 1: 342 – 343 (reprinted in Current Contents 45: 9-13, November 11 1991). http://cogprints.org/1581/

    Harnad, S. (1996) Implementing Peer Review on the Net: Scientific Quality Control in Scholarly Electronic Journals. In: Peek, R. & Newby, G. (Eds.) Scholarly Publishing: The Electronic Frontier. Cambridge MA: MIT Press. Pp 103-118. http://cogprints.org/1692/

    Harnad, S. (1997) Learned Inquiry and the Net: The Role of Peer Review, Peer Commentary and Copyright. Learned Publishing 11(4) 283-292. Short version appeared in 1997 in Antiquity 71: 1042-1048. Excerpts also appeared in the University of Toronto Bulletin: 51(6) P. 12. http://cogprints.org/1694/

    Harnad, S. (1998/2000/2004) The invisible hand of peer review. Nature [online] (5 Nov. 1998), Exploit Interactive 5 (2000): and in Shatz, B. (2004) (ed.) Peer Review: A Critical Inquiry. Rowland & Littlefield. Pp. 235-242. http://cogprints.org/1646/

    Harnad S. (2002) BBS Valedictory Editorial (2002) Behavioral and Brain Sciences 24 http://users.ecs.soton.ac.uk/harnad/Temp/bbs.valedict.html

    Harnad, S. (2003) PostGutenberg Peer Review the invariant essentials and the newfound efficiencies http://users.ecs.soton.ac.uk/harnad/Temp/peerev.pdf

    Harnad, S. (2009) The PostGutenberg Open Access Journal. In: Cope, B. & Phillips, A (Eds.) The Future of the Academic Journal. Chandos. http://eprints.ecs.soton.ac.uk/15617/

  15. Albert Hoch says:

    Quit replacing the word “use” with “leverage”. Spoils your steel hard prose.

Comments are closed.