Posts Tagged Peer Review

Systematic Review claims acupuncture as effective as antidepressants: Part 1: Checking the past literature


A recent systematic review in PLOS One raised the question whether acupuncture and other alternative therapies are as effective as antidepressants and psychotherapy for depression. The authors concluded

 differences were not seen with psychotherapy compared to antidepressants, alternative therapies [and notably acupuncture] or active intervention controls

or put it differently,

antidepressants alone and psychotherapy alone are not significantly different from alternative therapies or active controls.

There are clear messages here. To consumers: Why take antidepressants with their long delay and uncertainty in showing any benefits–but immediate side effects and potential risks–when a few sessions of acupuncture work just as well? To promoters of acupuncture and alternative therapies: you can now cite an authoritative review in the peer-reviewed PLOS One as scientific evidence that your treatments is as effective as scary antidepressants and time-consuming psychotherapy when you make appeals to consumers and to third-party payers.

The systematic review had five co-authors, of whom three have been involved in previous meta-analyses of the efficacy of antidepressants. However, fourth author Irving Kirsch will undoubtedly be the author most recognizable to consumers and policymakers, largely because his relentless media campaign claiming antidepressants are essentially worthless, no better than placebo. For instance, in an interview with CBS 60 Minutes Irving Kirsch: The difference between the effect of a placebo and the effect of an antidepressant is minimal for most people.

Irving Kirsch: The difference between the effect of a placebo and the effect of an antidepressant is minimal for most people.

Posted in: Acupuncture, Neuroscience/Mental Health, Science and the Media

Leave a Comment (32) →

The problem with preclinical research? Or: A former pharma exec discovers the nature of science

If there’s one thing about quacks, it’s that they are profoundly hostile to science. Actually, they have a seriously mixed up view of science in that they hate it because it doesn’t support what they believe. Yet at the same time they very much crave the imprimatur that science provides. When science tells them they are wrong, they therefore often try to attack the scientific method itself or claim that they are the true scientists. We see this behavior not just in quackery but any time scientific findings collide with entrenched belief systems, for example, medicine, evolution, anthropogenic global warming, and many others. So it was not surprising that a rant I saw a few weeks ago by a well-known supporter of pseudoscience who blogs under the pseudonym of Vox Day caught my interest. Basically, he saw a news report about an article in Nature condemning the quality of current preclinical research. From it, he draws exactly the wrong conclusions about what this article means for medical science:

Fascinating. That’s an 88.6 percent unreliability rate for landmark, gold-standard science. Imagine how bad it is in the stuff that is only peer-reviewed and isn’t even theoretically replicable, like evolutionary biology. Keep that figure in mind the next time some secularist is claiming that we should structure society around scientific technocracy; they are arguing for the foundation of society upon something that has a reliability rate of 11 percent.

Now, I’ve noted previously that atheists often attempt to compare ideal science with real theology and noted that in a fair comparison, ideal theology trumps ideal science. But as we gather more evidence about the true reliability of science, it is becoming increasingly obvious that real theology also trumps real science. The selling point of science is supposed to be its replicability… so what is the value of science that cannot be repeated?

No, a problem with science as it is carried out by scientists in the real world doesn’t mean that religion is true or that a crank like Vox is somehow the “real” intellectual defender of science. Later, Vox doubles down on his misunderstanding by trying to argue that the problem in this article means that science is not, in fact, “self-correcting.” This is, of course, nonsense in that the very article Vox is touting is an example of science trying to correct itself. Be that at it may, none of this is surprising, given that Vox has demonstrated considerable crank magnetism, being antivaccine, anti-evolution, an anthropogenic global warming denialist, and just in general anti-science, but he’s not alone. Quackery supporters of all stripes are jumping on the bandwagon to imply that this study somehow “proves” that the scientific basis of medicine is invalid. A writer at Mike Adams’ wretched hive of scum and quackery,, crows:

Begley says he cannot publish the names of the studies whose findings are false. But since it is now apparent that the vast majority of them are invalid, it only follows that the vast majority of modern approaches to cancer treatment are also invalid.

But does this study show this? I must admit that it was a topic of conversation at the recent AACR meeting, given that the article was published shortly before the meeting. It’s also been a topic of e-mail conversations and debates at my very own institution. But do the findings reported in this article mean that the scientific basis of cancer treatment is so off-base that quackery of the sort championed by Mike Adams is a viable alternative or that science-based medicine is irrevocably broken?

Not so fast there, pardner…

Posted in: Basic Science, Cancer, Clinical Trials

Leave a Comment (16) →

Peer Review and the Internet

Peer-review has been the cornerstone of quality control in academia, including science and medicine, for the past century. The process is slow and laborious, but a necessary filter in order to maintain a certain standard within the literature. Yet more and more scholars are recognizing the speed, immediacy, and openness of the internet as a tool for exchanging ideas and information, and this is causing some to question the methods of peer review. A recent New York Times article discusses this issue.

This issue is very relevant to Science-Based Medicine as this is in part an experiment – an attempt to produce a high quality, editorially filtered, but not peer-reviewed, online journal. Our process here is simple. Outside submissions are reviewed by two or more editors and typically are either accepted with minor revisions or rejected. In addition we have a staff of regular contributors – those who have a proven track record of producing high quality articles. There is no pre-publication review for their submissions, and they are able to post directly to SBM.

Because many of the issues we cover are timely, we emphasize speed of publication. Therefore copy-editing is done post-publication – the notion being that our readers can tolerate a few typos in order to gain access to material more quickly.


Posted in: Medical Academia

Leave a Comment (15) →

Does peer review need fixing?

One of the most important aspects of science is the publication of scientific results in peer-reviewed journals. This publication serves several purposes, the most important of which is to communicated experimental results to other scientists, allowing other scientists to replicate, build on, and in many cases find errors in the results. In the ideal situation, this communication results in the steady progress of science, as dubious results are discovered and sound results replicated and built upon. Of course, scientists being human and all, the actual process is far messier than that. In fact, it’s incredibly messy. Contrary to popular misconceptions about science, it doesn’t progress steadily and inevitably. Rather, it progresses in fits and starts, and most new scientific discoveries go through a varying period of uncertainty, with competing labs reporting conflicting results. To achieve consensus about a new theory can take relatively little time (for example, the less than a decade that it took for Marshall and Warren’s hypothesis that peptic ulcer disease is largely caused by H. pylori or the relatively rapid acceptance of Einstein’s Theory of Relativity) to much longer periods of time.

One of the pillars of science has traditionally been the peer review system. In this system, scientists submit their results to journals for publication in the form of manuscripts. Editors send these manuscripts out to other scientists to review them and decide if the science is sound, if the methods appropriate, and if the conclusions are justified by the data presented. This step of the process is very important, because if editors don’t choose reviewers with the appropriate expertise, then serious errors in review may occur. Also, if editors choose reviewers with biases so strong that they can’t be fair, then science that challenges such reviewers’ biases may never see print in their journals. The same thing can occur to grant applications. In the NIH, for instance, the scientists running study sections must be even more careful in choosing scientists to be on their study sections and review grant applications, not to mention picking which scientists review which grants. Biases in reviewing papers are one thing; biases in reviewing grant applications can result in the denial of funding to worthy projects in favor of projects less worthy that happen to correspond to the biases of the reviewers.

I’ve discussed peer review from time to time, although perhaps not as often as I should. My view tends to be that, to paraphrase Winston Churchill’s invocation of a famous quote about democracy, peer review is the worst way to weed out bad science and promote good science, except for all the others that have been tried. One thing’s for sure, if there’s a sine qua non of an anti-science crank, it’s that he will attack peer review relentlessly, as HIV/AIDS denialist Dean Esmay did. Indeed, in the case of Medical Hypotheses, the lack of peer review let the cranks run free to the point where even Elsevier couldn’t ignore it any more. One thing’s for sure. Peer review may have a lot of defects and blindnesses, but lack of peer review is even worse. It’s no wonder why cranks of all stripes loved Medical Hypotheses.

Posted in: Medical Academia, Science and Medicine

Leave a Comment (42) →

Open-Access Peer Review: Increasing the Noise To Signal Ratio

Readers of Science Based Medicine are quite familiar with the distressingly common logical leap made by disgruntled healthcare consumers into alternative medicine. It goes something like this: I had a terrible experience with a doctor who [ignored/patronized/misdiagnosed] me and I also heard something horrible in the media about a pharmaceutical company’s misbehavior [hiding negative results/overstating efficacy/overcharging for medications], therefore alternative treatments [homeopathy/acupuncture/energy healing, etc.] must be more effective than traditional medicine.

Much to my dismay, a similar logical leap is being made about online health information. It goes something like this: Peer reviewing is biased and often keeps innovative research hidden to the world at large, therefore the best kind of peer review is open-access where anyone in the world can contribute.

You may feel free to slap your forehead now.

While I have absolutely no doubt that doctors have their shortcomings, and that some have created less than pleasant healthcare experiences for their patients – the solution to these shortcomings is not to dive headlong into snake oil. Moreover, I agree that the current peer-review process has its flaws and limitations – the solution is not to ask Aunt Enid in Omaha what she thinks of the recent meta-analysis of perioperative beta blockers in patients having non-cardiac surgery.

Peter Frishauf, the founder of Medscape, recently published a webcast editorial predicting that:

“Peer review as we know it will disappear. Rather than the secretive prepublication review process followed by most publishers today, including Medscape, most peer review will occur transparently, and after publication.”

He goes on to describe a Wikipedia-like review scenario where:

“Any user can start an article, link it to related sources, and publish revisions with a click of the mouse. Anyone who reads an article can edit it.”

I know and like Peter very much, and his foresight (that publishing should become open-access), combined with the leadership of editorial heavy-weight, Dr. George Lundberg, led to the creation of the first really successful, quality, free online medical journal. This was no small feat, and a sure victory for global medical education efforts.

But the reason for The Medscape Journal’s success is not the “democratization” of peer review – but the democratization of access to trustworthy information. The quality controls are still in place – and must remain so – otherwise its value as a peer-reviewed journal will be utterly lost. Who should trust the edits of unqualified readers? Should science be determined by popular vote? Should all research be published by journals, regardless of its fatal flaws?


Posted in: General, Science and Medicine, Science and the Media

Leave a Comment (10) →