Science-Based Medicine

“Ridiculing RCTs and EBM”

Last week Val Jones posted a short piece on her BetterHealth blog in which she expressed her appreciation for a well-known spoof that had appeared in the British Medical Journal (BMJ) in 2003:

Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials

Dr. Val included the spoof’s abstract in her post linked above. The parachute article was intended to be humorous, and it was. It was a satire, of course. Its point was to call attention to excesses associated with the Evidence-Based Medicine (EBM) movement, especially the claim that in the absence of randomized, controlled trials (RCTs), it is not possible to comment upon the safety or efficacy of a treatment—other than to declare the treatment unproven.

A thoughtful blogger who goes by the pseudonym Laika Spoetnik took issue both with Val’s short post and with the parachute article itself, in a post entitled #NotSoFunny – Ridiculing RCTs and EBM.

Laika, whose real name is Jacqueline, identifies herself as a PhD biologist whose “work is split 75%-25% between two jobs: one as a clinical librarian in the Medical Library and one as a Trial Search Coordinator (TSC) for the Dutch Cochrane Centre.” In her post she recalled an experience that would make anyone’s blood boil:

I remember it well. As a young researcher I presented my findings in one of my first talks, at the end of which the chair killed my work with a remark that made the whole room of scientists laugh, but was really beside the point…

This was not my only encounter with scientists who try to win the debate by making fun of a theory, a finding or …people. But it is not only the witty scientist who is to *blame*, it is also the uncritical audience that just swallows it.

I have similar feelings with some journal articles or blog posts that try to ridicule EBM – or any other theory or approach. Funny, perhaps, but often misunderstood and misused by “the audience”.

Jacqueline had this to say about the parachute article:

I found the article only mildly amusing. It is so unrealistic, that it becomes absurd. Not that I don’t enjoy absurdities at times, but absurdities should not assume a life of their own.  In this way it doesn’t evoke a true discussion, but only worsens the prejudice some people already have.

Continue Reading »

As promised at the meeting. Let me know by comment if you think I left anything out.

Continue Reading »

Homeopathy and Science

This week’s entry† is a summary of some of the tests of homeopathy. It is a necessary prelude to a discussion of how homeopaths and their apologists promote the method. Several tenets of homeopathy lend themselves to tests. The doctrine of similia similibus curantur (“like cures like”) was tested by Hahnemann himself, as introduced in Part I of this blog. It is a special case that will be discussed further below. Hahnemann’s second doctrine, “infinitesimals,” suggests laboratory, animal, and clinical studies looking for specific effects of homeopathic preparations.

“Provings,” also called “homeopathic pathogenic trials,” suggest testing “provers” for the ability to distinguish between homeopathic preparations and placebos, and suggest asking homeopaths to identify specific remedies solely by the “symptoms” they elicit in “provers.” The homeopathic interview and prescribing scheme, gathering copious “symptoms” and matching them to the appropriate “remedy” in the Materia Medica, suggests testing homeopaths for consistency in symptom interpretations and prescriptions. The clinical practice suggests outcome studies, both of individual “conditions” (with the caveat that, strictly speaking, homeopathy does not recognize disease categories—only “symptom” complexes) and of the practice as a whole.

Several of these categories overlap. Several have been tested: the results have overwhelmingly failed to confirm homeopathy’s claims. I will mention a few of the more conspicuous examples.

Continue Reading »

After the previous posting on the Bayesian approach to clinical trial data, several new comments made it clear to me that more needed to be said. This posting addresses those comments and adds a few more observations regarding the unfortunate consequences of EBM’s neglect of prior probability as it applies to “complementary and alternative medicine” (“CAM”).†

The “Galileo Gambit” and the Statistics Gambit

Reader durvit wrote:

A very interesting example, for a number of people, might be estimating the prior probability for Marshall and Warren’s early work on Helicobacter pylori and its impact on gastroduodenal management. I frequently have Marshall quoted to me as a variation on the Galileo gambit, so establishing whether he and Warren would have been helped or hindered by Bayesian techniques would be useful.

This suggestion raises a couple of issues. First, the “Galileo gambit” regarding Marshall and Warren’s discovery is a straw man (as durvit seems to have surmised). Continue Reading »

This is the third post in this series*; please see Part II for a review. Part II offered several arguments against the assertion that it is a good idea to perform efficacy trials of medical claims that have been refuted by basic science or by other, pre-trial evidence. This post will add to those arguments, continuing to identify the inadequacies of the tools of Evidence-Based Medicine (EBM) as applied to such claims.

Prof. Simon Replies

Prior to the posting of Part II, statistician Steve Simon, whose views had been the impetus for this series, posted another article on his blog, responding to Part I of this series. He agreed with some of what both Dr. Gorski and I had written:

The blog post by Dr. Atwood points out a critical distinction between “biologically implausible” and “no known mechanism of action” and I must concede this point. There are certain therapies in CAM that take the claim of biological plausibility to an extreme. It’s not as if those therapies are just implausible. It is that those therapies must posit a mechanism that “would necessarily violate scientific principles that rest on far more solid ground than any number of equivocal, bias-and-error-prone clinical trials could hope to overturn.” Examples of such therapies are homeopathy, energy medicine, chiropractic subluxations, craniosacral rhythms, and coffee enemas.

The Science Based Medicine site would argue that randomized trials for these therapies are never justified. And it bothers Dr. Atwood when a systematic review from the Cochrane Collaboration states that no conclusions can be drawn about homeopathy as a treatment for asthma because of a lack of evidence from well conducted clinical trials. There’s plenty of evidence from basic physics and chemistry that can allow you to draw strong conclusions about whether homeopathy is an effective treatment for asthma. So the Cochrane Collaboration is ignoring this evidence, and worse still, is implicitly (and sometimes explicitly) calling for more research in this area.

On the other hand:

There are a host of issues worth discussing here, but let me limit myself for now to one very basic issue. Is any research justified for a therapy like homeopathy when basic physics and chemistry will provide more than enough evidence by itself to suggest that such research is futile(?) Worse still, the randomized trial is subject to numerous biases that can lead to erroneous conclusions.

I disagree for a variety of reasons.

Continue Reading »

Introduction

“Either homeopathy works or controlled trials don’t!”

—Scottish homeopath David Reilly at the 2001 Harvard Medical School Complementary and Integrative Medicine Conference.

Reilly based that assertion on his own series of four small studies of homeopathic treatments of hay fever, asthma, and allergic rhinitis, the outcomes of which had been inconsistent and largely subjective. (1) Later he explained that small-minded skeptics in “conventional medicine” assume “homeopathy doesn’t work because it can’t work,” a view echoed by conference host Dr. David Eisenberg, then the Director of the Center for Alternative Medicine Research and Education at Harvard Medical School (now of the Osher Center); these comments were met with appreciative laughter from the partisan audience. If such charges were valid, it would indeed be fortunate that Harvard Medical School, several other medical schools, and the National Center for Complementary and Alternative Medicine (NCCAM) are promoting homeopathy, both as a clinical method and as a topic worthy of research.

Continue Reading »

“Symptoms,” Continued

Part II of this blog† introduced the homeopathic understanding of “symptoms” as they pertain both to “provings” in healthy subjects (now called “homeopathic pathogenic trials” or “HPTs”) and to histories elicited from patients. Hahnemann conflated “symptoms” and every random itch, ache, pain, sniffle, feeling, thought, dream, pimple or other sign, and anything else that might occur to a subject or a patient. This was amply demonstrated by Oliver Wendell Holmes, Sr., who seemed to doubt that such a morass would yield useful information. As unlikely as it may seem, today’s homeopaths are every bit as whimsical in their elicitation of “symptoms” as was Hahnemann.
Continue Reading »

Consider these statements:

…there is an evidence base for biofield therapies. (citing the Cochrane Review of Touch Therapies)

The larger issue is what constitutes “pseudoscience” and what information is worthy of dissemination to the public. Should the data from our well conducted, rigorous, randomized controlled trial [of 'biofield healing'] be dismissed because the mechanisms are unknown or because some scientists do not believe in the specific therapy?…Premature rejection of findings from rigorous randomized controlled trials are as big a threat to science as the continuation of falsehoods based on belief. Thus, as clinicians and scientists, our highest duty to patients should be to investigate promising solutions with high benefit/risk ratios, not to act as gatekeepers of information based on personal opinion.

–Jain et al, quoted here

…

Touch therapies may have a modest effect in pain relief. More studies on HT and Reiki in relieving pain are needed. More studies including children are also required to evaluate the effect of touch on children.

Touch Therapies are so-called as it is believed that the practitioners have touched the clients’ energy field.

It is believed this effect occurs by exerting energy to restore, energize, and balance the energy field disturbances using hands-on or hands-off techniques (Eden 1993). The underlying concept is that sickness and disease arise from imbalances in the vital energy field. However, the existence of the energy field of the human body has not been proven scientifically and thus the effect of such therapies, which are believed to exert an effect on one’s energy field, is controversial and lies in doubt.

—Cochrane Review of Touch Therapies, quoted here

 …

Science is advanced by an open mind that seeks knowledge, while acknowledging its current limits. Science does not make assertions about what cannot be true, simply because evidence that it is true has not yet been generated. Science does not mistake absence of evidence for evidence of absence. Science itself is fluid.

—David Katz

…

When people became interested in alternative medicines, they asked me to help out at Harvard Medical School. I realized that in order to survive there, one had to become a scientist. So I became a scientist.

—Ted Kaptchuk, quoted here.

…

 …It seems that the decision concerning acceptance of evidence (either in medicine or religion) ultimately reflects the beliefs of the person that exist before all arguments and observation.

 —Ted Kaptchuk, quoted here.

Together they betray a misunderstanding of science that is common not only to “CAM” apologists, but to many academic medical researchers. Let me explain. Continue Reading »

This is an addendum to my previous entry on Bayesian statistics for clinical research.† After that posting, a few comments made it clear that I needed to add some words about estimating prior probabilities of therapeutic hypotheses. This is a huge topic that I will discuss briefly. In that, happily, I am abetted by my own ignorance. Thus I apologize in advance for simplistic or incomplete explanations. Also, when I mention misconceptions about either Bayesian or “frequentist” statistics, I am not doing so with particular readers in mind, even if certain comments may have triggered my thinking. I am quite willing to give readers credit for more insight into these issues than might be apparent from my own comments, which reflect common, initial difficulties in digesting the differences between the two inferential approaches. Those include my own difficulties, after years of assuming that the “frequentist” approach was both comprehensive and rational—while I had only a cursory understanding of it. That, I imagine, placed me well within two standard deviations of the mean level of statistical knowledge held by physicians in general.

Continue Reading »

NB: This is a partial posting; I was up all night ‘on-call’ and too tired to continue. I’ll post the rest of the essay later…

Review

This is the fourth and final part of a series-within-a-series* inspired by statistician Steve Simon. Professor Simon had challenged the view, held by several bloggers here at SBM, that Evidence-Based Medicine (EBM) has been mostly inadequate to the task of reaching definitive conclusions about highly implausible medical claims. In Part I, I reiterated a fundamental problem with EBM, reflected in its Levels of Evidence scheme, that although it correctly recognizes basic science and other pre-clinical evidence as insufficient bases for introducing novel treatments into practice, it fails to acknowledge that they are necessary bases. I explained the difference between “plausibility” and “knowing the mechanism.”

I showed, with several examples, that in the EBM lexicon the word “evidence” refers almost exclusively to the results of clinical trials: thus, when faced with equivocal or no clinical trials of some highly implausible claim, EBM practitioners typically declare that there is “not enough evidence” to either accept or reject the claim, and call for more trials—although in many cases there is abundant evidence, other than clinical trials, that conclusively refutes the claim. I rejected Prof. Simon’s assertion that we at SBM want to “give (EBM) a new label,” making the point that we only want it to live up to its current label by considering all the evidence. I doubted Prof. Simon’s contention that “people within EBM (are) working both formally and informally to replace the rigid hierarchy with something that places each research study in context.”

In Part II I responded to the widely held assertion, also held by Prof. Simon, that there is “societal value in testing (highly implausible) therapies that are in wide use.” I made it clear that I don’t oppose simple tests of basic claims, such as the Emily Rosa experiment, but I noted that EBM reviewers, including those employed by the Cochrane Collaboration, typically ignore such tests. I wrote that I oppose large efficacy trials and public funding of such trials. I argued that the popularity gambit has resulted in human subjects being exposed to dangerous and unethical trials, and I quoted language from ethics treatises specifically contradicting the assertion that popularity justifies such trials. Finally, I showed that the alleged popularity of most “CAM” methods—as irrelevant as it may be to the question of human studies ethics—has been greatly exaggerated.

Continue Reading »