On the dangers of using valid placebo controls in clinical trials of acupuncture

ResearchBlogging.orgI don’t recall if I’ve ever mentioned this before on this blog, but there was a time when I was less skeptical of acupuncture than I am now. It’s true. Don’t get me wrong, though. I never for a minute considered that the whole rigamarole about “unblocking” or “redirecting” the flow of that mystical life force known as qi had anything to do with whether or not acupuncture did or did not have efficacy treating disease or other conditions. That was clearly a holdover from the pre-scientific medicine times in which most beliefs about the causes of disease involved either the wrath of the gods or vitalism, the latter of which is, when you come right down to it, the philosophical basis upon which many “complementary and alternative” (CAM) modalities are based, especially the so-called “energy healing” modalities, such as reiki, therapeutic touch, and, of course, acupuncture.

However, because unlike so many other “energy healing” methods, acupuncture involved an actual physical action upon the body, namely the insertion of thin needles into the skin to specified depths, it did not seem to me entirely unreasonable that there might be some sort of physiological effect that might produce a therapeutic result. At least, that’s what I used to think until I actually started paying attention to the scientific literature on acupuncture. That’s when I started to realize that “there’s no ‘there’ there,” if you know what I mean. Horribly designed studies with either no controls or utterly inadequate controls tend to be the norm in the acupuncture “literature” (if you can call it that). Moreover, acupuncture was touted as having value for conditions and procedures for which there is no plausible (or even mildly plausible) physiological mechanism by which it could be reasonably postulated to have an effect. Arthritis, allergies, headache, back pain, gastroesophageal reflux disease (GERD), Parkinson’s disease, post-operative nausea, hot flashes in breast cancer patients caused by the anti-estrogen drugs they have to take, infertility, it doesn’t matter. Seemingly acupuncture can do it all; it’s the Swiss Army knife of CAM therapies. Moreover, the “explanations” given to explain “how acupuncture works” seemed increasingly less plausible to me. Most of these explanations involve counterirritation or the release of opioids, and I’ve had an increasingly hard time believing that, even if these mechanisms are at play, they could have anything other than nonspecific effects, with no mechanism to explain how acupuncture could possibly do all things attributed to it. One rule of medical skepticism is that you should be very skeptical of modalities that are touted to be useful for a wide variety of medical conditions that have very different pathophysiology. Indeed, a funny thing happens when rigorous placebo controls are introduced, and that’s sometimes the placebo control does better than the “true” acupuncture; i.e., the evidence for acupuncture, taken in its totality, is completely compatible with placebo effect.

As Harriet Hall put it in her excellent analysis of a study purporting to show that acupuncture is useful for GERD:

This study falls into the category of what I call Tooth Fairy science. You could measure how much money the Tooth Fairy leaves under the pillow, whether she leaves more cash for the first or last tooth, whether the payoff is greater if you leave the tooth in a plastic baggie versus wrapped in Kleenex. You can get all kinds of good data that is reproducible and statistically significant. Yes, you have learned something. But you haven’t learned what you think you’ve learned, because you haven’t bothered to establish whether the Tooth Fairy really exists.

One area that acupuncturists keep needling away at is infertility. Somehow, in the CAM community it’s become conventional wisdom that acupuncture can somehow increase the chance of success for couples undergoing in vitro fertilization (IVF). Indeed, early this year I wrote about a meta-analysis that concluded that acupuncture did actually increase the success rate of IVF and why it did not show what its authors thought it showed. Tooth fairy science, indeed.

Now comes yet another study being publicized in the media that examines once again the question of whether sticking needles into women before they undergo IVF can increase their chances of conceiving. I could not help but be extremely amused by the title given to the ScienceDaily story about it: Placebo Acupuncture Is Associated With Higher Pregnancy Rate After IVF Than Real Acupuncture. You have to love a headline like that, and opening paragraphs like this:

A study comparing the effects of real and placebo acupuncture on pregnancy rates during assisted reproduction has found that, surprisingly, placebo acupuncture was associated with a significantly higher overall pregnancy rate than real acupuncture.

The study, published online in the journal Human Reproduction, looked at real and placebo acupuncture given on the day of embryo transfer in 370 patients in a randomised, double blind trial (where neither the patients nor the doctors knew which treatment was being given).

The researchers found that the overall pregnancy rate (defined by a positive urinary pregnancy test) for placebo acupuncture was 55.1%, versus 43.8% for the real acupuncture.

I bet that result caused some head-scratching.

This reminds me of a study not too long ago that found a similar–shall we say?–disconcerting result in which acupuncture was studied for its ability to relieve arm pain from repetitive use injuries, which was funded, amusingly enough, by the National Center for Complementary and Alternative Medicine (NCCAM). Naturally, I had to go straight to the source. Fortunately, you, too, can do the same because the manuscript is available online for free for everyone, unlike most such papers, which are hidden behind a subscription firewall, not easily accessible by any but academics. The study, coming out of China, is entitled A randomized double blind comparison of real and placebo acupuncture in IVF treatment, and its lead investigator is Dr. Ernest Hung Yu Ng, Associate Professor in the Department of Obstetrics and Gynaecology at the University of Hong Kong (People’s Republic of China. So, let’s take a look at this study.

The hypothesis, not surprisingly, is that acupuncture could improve the efficacy of IVF and result in a higher pregnancy rate. The specific acupuncture regimen consisted of two 25-minute sessions, one before embryo transfer and one after embryo transfer was completed. Patient were informed of the study on the day of transvaginal ultrasound-guided oocyte retrieval (TUGOR), at which time a practitioner of traditional Chinese medicine (TCM) “diagnosed” thusly subjects who agreed to participate in the study:

A registered traditional Chinese medicine (TCM) practitioner (E.W.S.S.) categorized these patients’ conditions by the four diagnostic methods: observation, auscultation, interrogation and palpation according to the TCM principles (Maciocia, 1998). They were classified into the related syndromes which included Kidney Yang/Yin deficiency, Liver Qi stagnation with blood stasis, Spleen Qi deficiency with Phlegm and combination of those syndromes.

I wonder if spleen qi deficiency requires a splenectomy or if we can prescribe more yang or yin for a Yang/Yin deficiency. Sorry, couldn’t resist.

Patients were randomized to receive either “real” acupuncture of “sham” (a.k.a. placebo) acupuncture, and the placebo acupuncture needle was described thusly:

The Streitberger’s placebo needles (Streitberger and Kleinhenz, 1998; Asiamed, Pullach, Germany; 0.30 x 30 mm) were used. The appearance of the placebo needle was the same as the needle used in the real acupuncture group. The placebo needle was not fixed into the copper handle and the tip of the needle was blunt. When it was pushed forward against the skin, the needle slid into the handle and the whole needle appeared shortened. This also gave patients a pricking penetration sensation. To place the placebo needle in position, the Park’s placebo device (Park et al., 1999; Dong Bang AcuPrime, Exeter, UK) was used. The same acupoints and procedures as in the real acupuncture group were applied.

Overall, the design of the trial appears pretty sound. It was randomized and double-blinded to the extent possible with a treatment like acupuncture, and, as all good double-blind studies do, patients were asked afterward to guess whether they were in the treatment group or the placebo group in order to assess the quality of blinding. The study was powered to detect a 15% increase in the rate of pregnancy based on the clinic’s usual success rate of 35%, which required 370 patients. (Note how it was assumed that acupuncture would increase the success rate of IVF.) Overall, on first glance, the two groups seemed to be comparable, although I was a bit disturbed to see that the statistics showing that they were, in fact, comparable in all the parameters measured were not included in the table summarizing the two groups. My other quibble is that I was a bit surprised that the primary outcome the investigators chose to look at was pregnancy rates as assessed by urine pregnancy tests. As anyone who’s ever undergone IVF knows, not all positive pregnancy tests result in an ongoing pregnancy or a live birth, not by a longshot. When discussing the success rate of IVF, what patients are interested in is the live birth rate. Anything less is a failure. Looking at the table of the results, I think I see why the authors were more interested in the overall pregnancy rate. It is the only outcome measure the produced a statistically significant difference between the two groups.

Let’s take a closer look. The overall pregnancy rate (a positive urine pregnancy test) was 43.8% for the “real” acupuncture group and 55.1% for the placebo acupuncture group (p =0.038). However, the numbers become less and less different as we go deeper into measurements of pregnancy that matter more. For example, generally after a round of IVF that results in a positive pregnancy test, the first big hurdle afterward is the 4 week ultrasound that is done to see if there is a fetal sac, and ongoing pregnancy is usually defined as a pregnancy resulting in a fetal heartbeat detectable by ultrasound at 8 weeks. The definition of live birth is fairly self-explanatory, although for purposes of this study it was defined as a live birth occurring after 24 weeks. In any case, clinical pregnancy rates for “real” acupuncture and placebo acupuncture were 38.9% and 49.2%, respectively (p=0.059); ongoing pregnancy rates, 31.9% and 40.5% (p=0.105); live birth rates, 29.7% and 38.4% (p=0.100). There was similarly no difference between implantation rates, miscarriage rates, or ectopic pregnancy rates. In fact, this study is not a study showing that acupuncture does anything to improve or harm success rates for IVF. It’s a negative study. There is no statistically significant difference between the outcomes in the only measure that patients really care about, live births. Indeed, there was in addition no difference between the groups in endometrial or subendometrial vascular density, nor were there any statistically significant differences in serum cortisol levels. Add to that the fact that the authors appear not to have controlled for multiple comparisons using an appropriate statistical correction (remember, the more comparisons one makes, the more likely there is to be a statistically “significant” difference by chance alone), and my conclusion is that this study would show us absolutely no difference between the groups if the appropriate statistical correction were applied.

That doesn’t stop the authors from doing all sorts of contortions over the results in the Discussion section. For example, the authors did find a significant decrease in serum cortisol after both “real” and placebo accupuncture, as well as an improvement in uterine blood flow parameters. To me this suggests decreased stress in both groups more than anything else, entirely compatible with the placebo effect. What’s more interesting is how the authors try to explain their result after first expressing their surprise in the discussion:

We found a significantly higher overall pregnancy rate following placebo acupuncture when compared with that of real acupuncture. There was a trend of higher rates of clinical pregnancy, ongoing pregnancy, live birth and implantation in the placebo acupuncture group, although the differences did not reach statistical significance. Surprisingly, significant changes in endometrial and subendometrial vascularity, serum cortisol concentration and the anxiety level were documented following both real and placebo acupuncture.

Why surprising? After all, this is exactly what would be predicted by the hypothesis that acupuncture acts primarily as a placebo; specifically, no differences in primary outcomes (ongoing pregnancy rates and live birth rates) but possibly differences in measurements related to patient anxiety, such as cortisol levels and uterine blood flow. No surprise there, unless you believe in acupuncture. The authors do go on to make a point that I actually somewhat agree with. Specifically, they lament that they didn’t include a no-acupuncture group as a control. Of course, the problem with adding such a group is that it would make it impossible to blind the study subjects in the no-acupuncture control group to their experimental group assignment and thus render this study no longer double blind. That is not an insignificant problem. However, it might be interesting to do in order to estimate the magnitude of the placebo effect.

More amusingly, Dr. Ng went into some contortions to try to explain his “surprising” result in an interview. In essence, he likened placebo acupuncture to acupressure and then waved his hands thusly:

Dr Ng gave two possible explanations for the results: “Placebo acupuncture is similar to acupressure and therefore is good enough to improve the pregnancy rate. Or else, it’s possible that real acupuncture may, in some way, reduce the pregnancy rate of acupuncture.

“So far there is no evidence that real acupuncture would adversely affect IVF outcomes because, in a previous meta-analysis of several acupuncture studies, the pregnancy rate was higher in the acupuncture groups than in the control groups. However, we cannot draw a firm conclusion about this from our current study as we did not compare the two groups with a third control group patients who received neither forms of acupuncture. Further studies should be conducted to compare placebo or non-invasive acupuncture and controls without acupuncture.”

You can sense the desperation. At least, I can, especially looking at his rationalization for the latter explanation, namely that “real” acupuncture somehow reduces the pregnacy rate. Of course, such a result would in fact be interesting if true, because it would indicate a physiological effect due to “real” acupuncture distinguishable from placebo acupuncture, just a negative one. In reality, the best explanation for this result is that there is no difference between the two groups (there isn’t, in fact) combined with a random finding of a seemingly “statistically significant” difference in one parameter, namely the overall pregnancy rate as measured by the rate of positive urine pregnancy tests in each subject group. Combine that with a hypothesis with a very low prior probability, which significantly increases the likelihood of a false positive result, as John Ioannidis has shown, and there’s no real difficulty in explaining Dr. Ng’s result. The only thing unusual in this case is that the false “positive” result was a “positive” in the wrong direction. Alas for poor Dr. Ng, he was tripped up by the vagaries of comparing two different (and almost certainly equal) placebos against each other. Do such studies long enough, and inevitably sometimes the “wrong” placebo will win.


E. W. S. So, E. H. Y. Ng, Y. Y. Wong, E. Y. L. Lau, W. S. B. Yeung, P. C. Ho (2008). A randomized double blind comparison of real and placebo acupuncture in IVF treatment Human Reproduction DOI: 10.1093/humrep/den380

Posted in: Acupuncture, Clinical Trials, Medical Academia, Science and the Media

Leave a Comment (16) ↓

16 thoughts on “On the dangers of using valid placebo controls in clinical trials of acupuncture

  1. daedalus2u says:

    This result is completely consistent with my understanding of the physiology behind the placebo effect as being the neurogenic triggering of a “standing down” from a higher stress state of “fight or flight”. A state of high stress is a bad time to get pregnant, so it would make perfect sense that a placebo (by shifting physiology away from a high stress state) would make getting pregnant easier.

    The difference between sham acupuncture and real acupuncture has a simple explanation, sham acupuncture can be a better placebo because it doesn’t involve damaging the skin by piercing it with a needle. How good a placebo sham acupuncture and real acupuncture are depends on the hoodoo skill of the shaman administering either of them.

    What this study does say is that real acupuncture makes IVF less successful and should not be used. It is unethical to use real acupuncture as part of IVF.

  2. mjranum says:

    Whenever someone starts touting accupuncture around me, I ask them “can you use accupuncture to get high?”
    “Yeah, you know, wasted? I mean, you can supposedly adjust practically any internal state with accupuncture – why not give me a healthy simulation of an alcohol high or maybe adjust my ch’i so that I’ve got an equivalent effect to eating some ecstacy?”

    Of course, I’m kidding. But here’s the truth: if accupuncture DID work, people would be using it to kill eachother, get wasted, or give themselves erections on demand. The reason they don’t is simple: accupuncture doesn’t work.

  3. Fifi says:

    mjranum – That is a most excellent point…so to speak.

  4. tgbarton says:

    When are we going to create a drug that chemically triggers placebo effects for people even in a coma? Not only would it be a block-buster drug, it would put alt med out of business and be invaluable in developing all drugs.

    All you researching scientific types: get on that!

  5. wertys says:

    There is now sufficient evidence from well-controlled studies to justify me quitting medicine and getting trained in sham acupuncture. It’s more effective than ‘real’ acupuncture and you don’t need to be licensed or trained.

    I so love how these investigators all have total David Brent moments when asked about the results of their studies.

  6. daedalus2u says:

    tgbarton, I have that “drug” already. That is what my NO producing bacteria do. They trigger the placebo effect pharmaceutically.

  7. ingridity says:

    In response to: mjranum who said…
    “…But here’s the truth: if accupuncture DID work, people would be using it to kill eachother, get wasted, or give themselves erections on demand. The reason they don’t is simple: accupuncture doesn’t work….”

    Your simple conclusion is entirely too simple. Acupuncture is a very delicate and precise procedure, one by which you need many many years of training in order to learn how to do it properly to achieve results.

    The reasons for people not doing it to themselves to get “high” is that very few people know how to do this, and those who do, have no interest in that kind of silliness. If you just want to get high, there are much MUCH easier paths to pursue than spending years of training in Chinese medicine. Not to mention, it may not actually be one of the things acupuncture does for people – I don’t know enough about it to make that call.

    It’s like saying … if your doctor knows what he’s doing, then why doesn’t he get himself high and wasted and give himself errections all the time? Since he’s not, I guess he doesn’t REALLY know what he’s doing.


  8. ingridity says:

    I completely agree with Dr. Ng that this study doesn’t tell us much in that there was no true “control” group -I mean, why did they not use a control group of women who received no form of accupuncture or even accupressure (from the “fake” acupuncture.) This seems really weird to me that they didn’t compare this to women who received zero stimulation on their skin…

  9. wertys says:

    I’ll tell you why ingridity, they aren’t doing science. that’s why..

  10. whatdoyouthink says:

    Hi ingridity:

    I am curious why do you contend that a no treatment control group was necessary?

  11. Dickinson13 says:

    While I enjoyed reading this analysis, and agree with the lovely description of tooth fairy science, I was a little disturbed that the writer regarded Hong Kong as being a part of the Republic of China.

    Having worked on Hong Kong for five years, I think it is important to realise that Hong Kong SAR, while technically now part of China has a very different academic background: basically British, rather than Chinese. There are two medical schools: Hong Kong U, and Chinese University of HK, each of high quality. While one must be sceptical of any data that come out of mainland china, one can generally treat Hong Kong data with the same degree of respect for accuracy as that coming out of most other developed countries with reasonable standards: the US for example. That is not, of course to believe all the interpretation: as one does not with US papers.

    A mainlander would be under great moral pressure to ensure that their results conformed to the official line supporting Chinese traditional medicine: and sadly, this does seem to happen. A Hong Kong University staff doctor may be under some moral pressure to reach a conclusion favourable to traditional Chinese medicine, if he wants to get more funding for this type of work, but is expected to report the data properly. After all the very fact that the data in this paper is open enough to perform the entertaining surgical dissection done here makes the point.

    Jim Dickinson
    Formerly Professor of Family Medicine, at Chinese University of Hong Kong
    Hong Kong Special Administrative Region of China.

  12. David Gorski says:

    I understand that historically Hong Kong was more part of Britain than China, but politically it is now a part of People’s Republic of China, is it not? Wasn’t it turned back over to China in 1997 or so?

  13. thehatinthecat says:

    I have a quick question on the topic of Sham acupuncture.
    “Sham acupuncture means that needles are inserted, but not in the proscribed locations” [Neorologica Blog]( when this is being done are they using meridian points that have nothing to do with the treatment? Or are they inserting in random spots on the body making sure to avoid traditional meridian spots? Or what?
    I’ve never found a clear explanation on this.

  14. Harriet Hall says:

    There are at least two kinds of sham acupuncture. One involves points that are supposedly in locations away from traditional acupuncture points and meridians. Another uses a retractable needle like a stage dagger.

  15. thehatinthecat says:

    @Harriet Hall
    Thank you.

    An interesting look placebo for acupuncture, though maybe not ethical, would be to take a TCM diagnosis and then use acupuncture points that would actually be contraindication to the TCM diagnosis and see what happens.

Comments are closed.