This is an addendum to my previous entry on Bayesian statistics for clinical research.† After that posting, a few comments made it clear that I needed to add some words about estimating prior probabilities of therapeutic hypotheses. This is a huge topic that I will discuss briefly. In that, happily, I am abetted by my own ignorance. Thus I apologize in advance for simplistic or incomplete explanations. Also, when I mention misconceptions about either Bayesian or “frequentist” statistics, I am not doing so with particular readers in mind, even if certain comments may have triggered my thinking. I am quite willing to give readers credit for more insight into these issues than might be apparent from my own comments, which reflect common, initial difficulties in digesting the differences between the two inferential approaches. Those include my own difficulties, after years of assuming that the “frequentist” approach was both comprehensive and rational—while I had only a cursory understanding of it. That, I imagine, placed me well within two standard deviations of the mean level of statistical knowledge held by physicians in general.
Archive for Clinical Trials
This is actually the second entry in this series;† the first was Part V of the Homeopathy and Evidence-Based Medicine series, which began the discussion of why Evidence-Based Medicine (EBM) is not up to the task of evaluating highly implausible claims. That discussion made the point that EBM favors equivocal clinical trial data over basic science, even if the latter is both firmly established and refutes the clinical claim. It suggested that this failure in calculus is not an indictment of EBM’s originators, but rather was an understandable lapse on their part: it never occurred to them, even as recently as 1990, that EBM would soon be asked to judge contests pitting low powered, bias-prone clinical investigations and reviews against facts of nature elucidated by voluminous and rigorous experimentation. Thus although EBM correctly recognizes that basic science is an insufficient basis for determining the safety and effectiveness of a new medical treatment, it overlooks its necessary place in that exercise.
This entry develops the argument in a more formal way. In so doing it advocates a solution to the problem that has been offered by several others, but so far without real success: the adoption of Bayesian inference for evaluating clinical trial data.
Patients with heartburn are often diagnosed with GERD (gastroesophageal reflux disease) and treated with a drug called a proton pump inhibitor (PPI) to reduce stomach acid production. It is pretty effective, but it doesn’t always work. When it doesn’t, standard practice has been to double the dose of PPI. Doubling the dose only improves symptoms in 20-25%. Most patients who fail the single dose turn out to have normal esophageal acid exposure, or “functional” heartburn. In other words, the symptoms appear to be due to something other than excess acid – so it really may not make much sense to double the PPI dose. What else could doctors try?
Hype over science: Does acupuncture really improve the chances of success for in vitro fertilization?
There it was on Friday greeting me on the ABC News website: “Study: Acupuncture May Boost Pregnancy” in bold blue letters, with the title of the webpage being “Needles Help You Become Pregnant.” The story began:
It sounds far-fetched sticking needles in women to help them become pregnant but a scientific review suggests that acupuncture might improve the odds of conceiving if done right before or after embryos are placed in the womb.
The surprising finding is far from proven, and there are only theories for how and why acupuncture might work. However, some fertility specialists say they are hopeful that this relatively inexpensive and simple treatment might ultimately prove to be a useful add-on to traditional methods.
By the end of the day, the story was all over the media, including radio, TV, news websites, the blogosphere, and various other outlets, all trumpeting the message that a scientific study says that acupuncture can help infertile couples conceive. Nary a skeptical word seemed to be found. Knowing very well just how far parents will go to conceive, I was curious: Did this study actually say what the media says it said? What was so new and radical about this study that it rated a press release and a lot of promotion? Do we here at SBM (particularly Steve) need to rethink our extreme skepticism about acupuncture, given the poor quality evidence and lack of even a glimmer of a convincing physiologic mechanism to explain its supposed activities?
News bulletin on BBC NEWS International version, 8 Feruary 2008:“Acupuncture ‘boosts IVF chances.’ Acupuncture may increase the success rates of fertility treatment, according to a study. “
(Manheimer E, Zhang G, Udoff L, Haramati A, Langenberg P, Berman BM, Bouter LM. Effects of acupuncture on rates of pregnancy and live birth among women undergoing in vitro fertilisation: systematic review and meta-analysis. BMJ. 2008 Feb 7)
First off, how plausible is the claim? The press release states that acupuncture had been used in China fior thousands of years for infertility. Has it? No medical historian writing I have seen made such an interpretation of ancient texts. Maybe I missed something…possible. But acupuncture was not used for specific disorders or purposes, but was used as a sort of panacea to cause balance of either the Yin and Yang or of the relationship of the individual with the 5 elements and the cosmos and the earth. There is nothing specific in claims of acupuncture in traditional Chinese Medicine history. Who gave the news people that misleading lead-in?
Second, what is the plausibility that acupuncture could possibly affect a laboratory procedure on tissue removed from the subject, regardless of timing? Negligible to none. There is no consistent and credible information that acupuncture is effective for anything, except as a conditiong agent for perception of symptoms.
So, does acupuncture increase the success of IVF?
Homeopathy and Science: Discussion, Summary and Conclusions
I was not surprised by a couple of the dissenting comments after Part IV of this blog. One writer worried that I had neglected, presumably for nefarious reasons, to cite replications of Benveniste’s results; another cited several examples of “positive” homeopathy studies that I had failed to mention. I answered some of those points here. I am fully aware of such “positive” reports, including those seeming to support Benveniste. I didn’t cite them, but not in some futile hope of concealing their existence from the watchful eyes of the readership. I also didn’t cite several “negative” reports, including an independent, disconfirming report of one of the claims of David Reilly, whose words began this series,* and the most recent of several reviews (referenced here) to conclude that “the clinical effects of homoeopathy are placebo effects.” I didn’t cite those reports for the same reasons that I didn’t cite the “positive” studies: they are mere footnotes to the overwhelming evidence against homeopathy.
To explain why, it will be necessary to discuss some of the strengths and weaknesses of the project known as “Evidence-Based Medicine.”
The National Center for Complementary and Alternative Medicine (NCCAM): Your tax dollars hard at work
What’s an advocate of evidence- and science-based medicine to think about the National Center for Complementary and Alternative Medicine, better known by its abbrevation NCCAM? As I’ve pointed out before, I used to be somewhat of a supporter of NCCAM. I really did, back when I was more naïve and idealistic. Indeed, as I mentioned before, when I first read Wally Sampson’s article Why NCCAM should be defunded, I thought it a bit too strident and even rather close-minded. At the time, I thought that the best way to separate the wheat from the chaff was to apply the scientific method to the various “CAM” modalities and let the chips fall where they may.
Two developments over the last several years have led me to sour on NCCAM and move towards an opinion more like Dr. Sampson’s. First, after its doubling from FY 1998-2003, the NIH budget stopped growing. In fact, adjusting for inflation, the NIH budget is now contracting. NCCAM’s yearly budget remains in the range of $121 million a year, for well over $1 billion spent since its inception as the Office of Alternative Medicine in 1993. Its yearly budget contains enough money to fund around 75 to 100 new five year R01 grants, give or take. In tight budgetary times my view is that it is a grossly irresponsible use of taxpayer money not to prioritize funding for projects that have hypotheses behind them that have a reasonable chance of being true. Scarce NIH funds should not be for projects that have as their basis hypotheses that are outlandishly implausible from a scientific standpoint. Second, I’ve seen over the last few years how NCCAM is not only funding research (most of which is of the sort that wouldn’t stand a chance in a study section from other Institutes or Centers)) but it’s funding training programs. Indeed, that was the core complaint against NCCAM: that it facilitates and promotes the infiltration of nonscience- and nonevidence-based treatments falling under the rubric of so-called “complementary and alternative” or “integrative” medicine into academic medicine. However, NCCAM cannot do otherwise, given its mission:
- Explore complementary and alternative healing practices in the context of rigorous science.
- Train complementary and alternative medicine researchers.
- Disseminate authoritative information to the public and professionals.
If, in fact, NCCAM actually did devote itself solely to “rigorous science” with regard to “alternative” healing practices, I would have much less problem with it than I do. However, it broadly interprets the second and third parts of its mission. For example, it views part of its mission as promotion, rather than study: “Supporting integration of proven CAM therapies. Our research helps the public and health professionals understand which CAM therapies have been proven to be safe and effective.” This would be all well and good if NCCAM had as yet actually proven any CAM therapies to be at least effective, but it has not. Worse, it has not even managed to demonstrate any of them to be ineffective, either, thus leading to endless studies of modalities that either do not work or at the very least would have marginal efficacy.
Still, I thought; All questions of promotion of CAM modalities aside, least there’s the science. Surely, under the auspices of the NIH, NCCAM must be funding some high-quality studies into CAM modalities that couldn’t be done any other way. That thought died when NCCAM announced last week the studies that it had funded during FY 2007.
Homeopathy and Science
This week’s entry† is a summary of some of the tests of homeopathy. It is a necessary prelude to a discussion of how homeopaths and their apologists promote the method. Several tenets of homeopathy lend themselves to tests. The doctrine of similia similibus curantur (“like cures like”) was tested by Hahnemann himself, as introduced in Part I of this blog. It is a special case that will be discussed further below. Hahnemann’s second doctrine, “infinitesimals,” suggests laboratory, animal, and clinical studies looking for specific effects of homeopathic preparations.
“Provings,” also called “homeopathic pathogenic trials,” suggest testing “provers” for the ability to distinguish between homeopathic preparations and placebos, and suggest asking homeopaths to identify specific remedies solely by the “symptoms” they elicit in “provers.” The homeopathic interview and prescribing scheme, gathering copious “symptoms” and matching them to the appropriate “remedy” in the Materia Medica, suggests testing homeopaths for consistency in symptom interpretations and prescriptions. The clinical practice suggests outcome studies, both of individual “conditions” (with the caveat that, strictly speaking, homeopathy does not recognize disease categories—only “symptom” complexes) and of the practice as a whole.
Several of these categories overlap. Several have been tested: the results have overwhelmingly failed to confirm homeopathy’s claims. I will mention a few of the more conspicuous examples.
Glucosamine and chondroitin, used separately or together, are among the more popular diet supplements. They are used widely for osteoarthritis, especially of the knee, and have been better studied than most other diet supplements. But do they really work?
The journal of my medical specialty, American Family Physician, recently published an article about the use of dietary supplements in osteoarthritis. They gave a “B” evidence rating to both glucosamine and chondroitin. This means there is inconsistent or limited-quality patient-oriented evidence. They recommended the use of glucosamine sulfate, saying, “Overall, the evidence supports the use of glucosamine sulfate for modestly reducing osteoarthritis symptoms and possibly slowing disease progression.” They did not exactly recommend chondroitin, although they said it “may provide modest benefit for some patients.”
Annals of Questionable Evidence: a new study reveals substantial publication bias in trials of anti-depressants
Part IV of the ongoing Homeopathy series will have to wait a day or two, because it is superceded by a recent, comment-worthy publication. Nevertheless, “H series” fans will find here a bit of grist for that mill, too.
An important role for this blog is to discuss problems of interpreting data from clinical studies. Academic medicine has committed itself, on the whole, to scientific rigor—to the extent that this is possible in messy, clinical (especially human) trials. Several tools have been proposed, and to a varying extent used, to enhance the rigor of clinical research and the reporting of clinical research. One of those tools is the registering of clinical trials prior to recruiting subjects. Registration would stipulate a trial’s a priori hypothesis(es), design, planned endpoints, and planned statistical methods, among other things. This would guard against several problems: publication bias—the tendency for some trials, usually “negative” ones, to go unreported; selective reporting of the results of a trial, if some are pleasing but others are not; and post hoc data analysis—finding data after the fact to suggest a novel hypothesis that will falsely be portrayed as an a priori hypothesis. Publication bias is also known as “selective publication” or the “file drawer problem”; post hoc analysis is also known as “data dredging” or “HARKing” (Hypothesizing After the Results are Known).
An article in the Jan. 17 issue of the New England Journal of Medicine demonstrates the usefulness of a trial registry:
Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy
Erick H. Turner, M.D., Annette M. Matthews, M.D., Eftihia Linardatos, B.S., Robert A. Tell, L.C.S.W., and Robert Rosenthal, Ph.D.