Chinese Systematic Reviews of Acupuncture

I’ll begin with the possibly shocking admission that I’m a strong supporter of the collection of ideas and techniques known as evidence-based medicine (EBM). I’m even the current President of the Evidence-Based Veterinary Medicine Association (EBVMA). This may seem a bit heretical in this context, since EBM  takes a lot of heat in this blog. But as Dr. Atwood has said, “we at SBM are in total agreement…that EBM “should not be without consideration of prior probability, laws of physics, or plain common sense,” and that SBM and EBM should not only be mutually inclusive, they should be synonymous.” So I have hope that by emphasizing the distinction between SBM and EBM and the limitations of EBM, we can engender the kind of changes in approach needed to address those limitations and eliminate the need for the distinction. One way of doing this is to critically evaluate the misuses of EBM in support of alternative therapies.

One of the highest levels of evidence in the hierarchy of evidence-based medicine is the systematic review. Unlike narrative reviews, in which an author selects those studies they consider relevant and then summarizes what they think the studies mean, which is a process subject to a high risk of bias, a systematic review identifies randomized controlled clinical trials according to an explicit and objective set of criteria established ahead of time. Predetermined criteria are also used to grade the studies evaluated by quality so any relationship between how well studies are conducted and the results can be identified. Done well, a systematic review gives a good sense of the balance of the evidence for a specific medical question.

Unfortunately, poorly done systematic reviews can create an strong but inaccurate impression that there is high-level, high-quality evidence in favor of a hypothesis when there really isn’t. Reviews of acupuncture research illustrate this quite well.

Acupuncture is one of the most studied practices in complimentary and alternative medicine (CAM), and this means there is a large volume of research to evaluate. While one might expect this to be a good thing, making it easier to tell whether acupuncture is effective for any specific medical problem, the amount of research studies actually makes for muddy waters in which the truth about the clinical efficacy of acupuncture is difficult to discern. The more studies there are, the greater the chance of getting some positive results even for an ineffective therapy. If the quality or methodology of the studies is poor, the results will be unreliable. And if numerous such studies of questionable quality exist, it becomes easier to generate systematic reviews which appear to provide high-level supporting evidence that doesn’t actually mean what it looks like it means.

For example, a recent systematic review of the use of acupuncture for pain following stroke appeared in the Journal of Alternative and Complementary Medicine.

Jung Ah Lee, Si-Woon Park, Pil Woo Hwang, Sung Min Lim, Sejeong Kook, Kyung In Choi, and Kyoung Sook Kang.  Acupuncture for Shoulder Pain After Stroke: A Systematic Review. The Journal of Alternative and Complementary Medicine. September 2012, 18(9): 818-823.

The conclusion seems quite promising; “It is concluded from this systematic review that acupuncture combined with exercise is effective for shoulder pain after stroke.” Given that a systematic review is high-level evidence, this ought to provide us with a fair degree of confidence that acupuncture is useful for this problem.

But a more detailed look casts a bit of doubt on this conclusion. For one thing, 453 studies were identified and only 7 met the quality criteria for inclusion. This suggests that, even in the eyes of acupuncture researchers, most acupuncture research is lousy. And the 7 studies that were chosen for evaluation were all conducted and published in China and all showed positive results. Their results may have as much to do with how research is conducted and published in China as with the efficacy of acupuncture for this problem.

While there is no question that some great scientific research is done in China, there is evidence for a systematic problem with the conduct and publication of alternative medicine studies there. Studies reported as randomized are  most often not actually properly randomized. And one review in 1998 found that  no negative study of acupuncture had ever been published in China. This strongly suggests that the acupuncture literature coming from China is unreliable due to poor methodological quality and a high risk of publication bias.

A review of systematic reviews published in the same journal as the review of acupuncture for shoulder pain also supports a skeptical interpretation of the first paper.

Bin Ma, Guo-qing Qi, Xiao-ting Lin, Ting Wang, Zhi-min Chen, and Ke-hu Yang.  Epidemiology, Quality, and Reporting Characteristics of Systematic Reviews of Acupuncture Interventions Published in Chinese Journals.  The Journal of Alternative and Complementary Medicine. September 2012, 18(9): 813-817.

These authors identified and evaluated systematic reviews of acupuncture research published in China and these were their findings:

Results: A total of 88 SRs were identified; none of the reviews had been updated. Less than one third (27.3%) were written by clinicians and one third (35.2%) were reported in specialty journals. The impact factor of 53.4% of the journals published was 0. Information retrieval was not comprehensive in more than half (59.1%) of the reviews. Less than half (36.4%) reported assessing for publication bias. Though 97.7% of the reviews used the term “systematic review” or “meta-analysis” in the title, no reviews reported a protocol and none were updated even after they had been published after 2 or more years.

Conclusions: Although many SRs of acupuncture interventions have been published in Chinese journals, the reporting quality is troubling. Thus, the most urgent strategy is to focus on increasing the standard of SRs of acupuncture interventions, rather than continuing to publish them in great quantity. This suggest that most systematic reviews of acupuncture published in China don’t search the literature thoroughly and don’t evaluate it properly. Given existing evidence that much of the research being reviewed is itself questionable, there is ample reason to be suspicious of the conclusions of such systematic reviews.

When supporters of acupuncture claiming to follow the principles of evidence-based medicine cite systematic reviews, there is a strong possibility that these reviews don’t actually fairly present the balance of the evidence. If they are poor quality reviews based on a biased sample of questionable studies, then they can only serve to create an inaccurate impression of the efficacy of acupuncture.

And there are  systematic reviews of the systematic reviews for acupuncture which have found that the balance of the evidence does not suggest a benefit from acupuncture: “In conclusion, numerous systematic reviews have generated little truly convincing evidence that acupuncture is effective in reducing pain.” A large number of studies makes it possible to generate high-level evidence both for and against a hypothesis, in this case concluding both that acupuncture does and does not relieve pain. That only further clouds the issue since naturally everyone cites those reviews which support their a priori position on acupuncture.

Another way of evaluating the state of the evidence on a given intervention is to compare the quality of studies with the likelihood of positive results. Dr. R. Barker Bausell has reviewed the acupuncture this way in his book  Snake Oil Science. As it turns out, the highest-quality studies of acupuncture consistently find  acupuncture works no better than placebo and that using  fake needles and  even jabbing the skin in random places with toothpicks work just as well as “real” acupuncture. Lower quality studies are more likely to be positive. This too sheds doubt on the reliability of positive clinical trials.

As supporters of acupuncture will undoubtedly point out, this doesn’t prove acupuncture doesn’t work in those conditions for which systematic reviews have stated it does work. It does show, however, that a lot of time, energy, and money has been spent on acupuncture research without generating a consistent body of evidence that can support it or justify any great confidence.

Which raises the issues of plausibility and prior probability, often cited as the primary sources of contention between between SBM and EBM. In theory, I do not object to clinical trial testing of interventions without well-established theoretical foundations. As  Sir Austin Bradford Hill, one of the early luminaries of clinical epidemiology, put it, “What is biologically plausible depends upon the biological knowledge of the day.” As CAM proponents delight in pointing out, sometimes wacky ideas prove true.

What they often fail to acknowledge, though, is that science does a pretty good job of accommodating such surprises if they can prove themselves through rigorous testing. The theory that Helicobacter could cause duodenal ulcers was considered implausible when proposed in 1982, and it won a Nobel prize for the proponents of the idea in 2005. That’s a pretty quick acceptance of an initially controversial idea, and it’s not consistent with the caricature of mainstream science as closed-minded and dogmatic.

In the real world, however, crazy ideas are far more likely to turn out to be wrong than revolutionary.  Dr. Sanden’s Electric Belt was at least as wacky as the idea that bacteria cause ulcers, but it has faded into history without any recognition from the Nobel committee. When time, money, and talent are limited (and they always are), spending them on ideas unlikely to bear fruit is hard to justify.

While a perfect world might allow for thorough, methodical testing of every possible practice, in this world we owe it to our patients to focus our energies on those ideas most likely to result in real help for them, those ideas which build on established knowledge rather than asking us to ignore or overturn it.

Finally, some sort of reasonable limit on the time and resources committed to investigating an idea is needed. When an adequate effort has been made and a strong, consistent body of evidence has failed to emerge, it is time to move on.

In the case of acupuncture, the original theoretical mechanisms invoked to explain why it should help (Ch’i, meridians, and so on) are vitalistic and inconsistent with established science. Attempts to find alternative mechanisms have yielded some interesting information about physiology and the mediation of pain sensation, but they have not turned up a coherent, unified theory of action supported by good evidence. And enormous numbers of clinical trials have been done over decades, again without yielding a consistent body of evidence supporting a specific therapeutic effect for acupuncture beyond the placebo effects of the therapeutic ritual.

So determining the truth about acupuncture requires more than simply looking for published systematic reviews. The quality of these reviews, and the studies they evaluate, must be critically appraised and the evidence at all levels, not simply clinical trials, must be considered. Finally, the proposed mechanisms by which acupuncture might work must also be critically evaluated to see if they are supported by good evidence and are not strongly at odds with established scientific knowledge. It is a misuse of evidence-based medicine to simply conduct poor quality systematic reviews on poor quality trials with a high risk of bias and then take the conclusion of these reviews at face value. A more comprehensive look at the question and the evidence at all levels is required. This is what is meant by science-based medicine, and it is what good evidence-based medicine should be.

Posted in: Acupuncture, Clinical Trials

Leave a Comment (28) ↓

28 thoughts on “Chinese Systematic Reviews of Acupuncture

  1. Janet says:

    One of my doctors has “evidence based medicine” as one of his areas of interest listed on his business card. I have been meaning to ask him what he means by this, but didn’t feel well-versed enough in the distinction (been meaning to search this site for clarification) to speak to this.

    I’m still confused. I get your point that EBM and SBM should be the same thing, but is that the accepted view among most EBM supporters, or are they mostly people looking for evidence of their pet modalities?

  2. geo says:

    ” I get your point that EBM and SBM should be the same thing, but is that the accepted view among most EBM supporters”

    The conclusions of well done EBM should be the same as well done SBM.

    I think that SBM’s emphasis upon plausibility can detract from the importance of looking carefully and sceptically at the evidence supporting all medical claims – plausible quackery can be even worse than implausible quackery. Coyne’s recent posts looking at the evidence to support claims about psychosocial interventions and cancer are a good example of the sort of work that needs to be done to improve EBM and SBM.

  3. EBM is a system for integrative controlled research evidence into the process of clinical decision making. It aims to replace opinion-based and faith-based medicine, shifting the emphasis for justifying practices from individual experience, tradition, and other subjective sources of information to the more reliable source of controlled research. It also involves critically evaluating the level and quality of research to help us give the most weight to the most reliable sources of data.

    The main distinction between SBM and EBM has to do with the degree to which one considers factors other than clinical trial data, such as the plausibility of a hypothesis. Strict EBM would say that we cannot, for example, dismiss homeopathy without extensive clinical trial evidence showing a lack of benefit, whereas SBM would say since it is theoretically impossible without a major re-write of well-established basic science principles, such trials are pointless and unethical.

    Such a distinction is important, but it is really quite small compared to the difference between SBM and EBM on the one hand, and faith-based medicine, which underlies most CAM, on the other.

  4. gaelyn37 says:

    I am concerned about a website which proposes to use evidence based medicine as the background database for their promotion of CAM. Do you have any experience or opinion about the validity of their approach?

  5. Bryan Bartens says:

    To sum up: EBM minus FBM-RCTs equals SBM?

  6. Harriet Hall says:

    I prefer EBM + CS = SBM

    CS stands for Common Sense

  7. pharmavixen says:

    Evidence-based medicine from one of the pioneers of the term:

    “The conscientious, explicit and judicious use of current best evidence in making decisions about the care of the individual patient. It means integrating individual clinical expertise with the best available external clinical evidence from systematic research.” (Sackett D, 1996)

    When I was in graduate school, one of the distinctions the professors made between evidence-based medicine and the way it was traditionally practiced was the emphasis on evidence and the downgrading of the role of expert opinion, because experts, however exalted their expertise, can have biases.

  8. Werdna says:

    In my mind EBM is Sackett’s et al’s attempt to produce some degree of formalism into evidence evaluation. It puts SR’s (with homogeneity) of RCT’s at the top, RCT’s in the middle and case reports at the bottom. I tend to liken this to how textual criticism changed the way people looked at ancient texts. It appeals to the parallel idea that evidence must be *weighed* not simply *counted*.

    SBM is the admission that in addition to this hierarchy there must also exist a higher standard of evidence. As EBM provides no facility for handling when SR’s conflict with chemistry, information theory or even each other. It’s these things which inform our priors.

  9. lilady says:

    This has to lead into NCCAM’s blog and the utterly ridiculous statements made by NCCAM’s director Dr. Josephine Briggs about “Prior Plausibility”…before NCCAM funds research into reiki, homeopathy, chiropractic and…acupuncture.

    Yes Dr. Briggs actually uses false analogies about treatments that were outside of “traditional” medicine to justify some of the ridiculous studies, funded by NCCAM:

    “Throughout the history of medicine, there are a number of examples of “quirky” ideas that encountered resistance from mainstream medicine, but eventually, through a combination of clinical experience and scientific pursuit, led to changes in health care. For example:

    Physical resistance training is good for people recovering from major physical trauma: Joseph Pilates, 1915

    Relaxation and breathing techniques help with the pain of childbirth: Fernand Lamaze, 1940

    Breastfeeding is good for babies, and mothers need help and support to establish successful
    breastfeeding: Edwina Froehlich, La Leche League founder, 1950s

    Extensive palliative support and reduced medical interventions should be provided to dying patients: Saunders, Wald, Kubler-Ross, 1960s

    So, sometimes good things come from challenges to mainstream orthodoxy. With that said, I do not advocate that we study every “quirky” idea that is proposed, but we must be willing to cast a critical eye over these ideas before dismissing them out-of-hand.”

    Many of the commenters on that post (including several who post on SBM), took her to task for her citing non-NCCAM-funded studies as “quirky ideas” and for her avoidance of “prior plausibility” while justifying the recently NCCAM-funded meta-analysis of acupuncture studies.

  10. ConspicuousCarl says:

    Harriet Hall on 12 Oct 2012 at 12:32 pm

    I prefer EBM + CS = SBM

    CS stands for Common Sense

    I agree with the intent, but “common sense” is too often used in a way that is similar to “intuition”. I try to avoid using that phrase in general because I am worried that people might take it the wrong way.

  11. Harriet Hall says:

    How about EBM + CT = SBM
    where CT stands for critical thinking.

  12. ConspicuousCarl says:

    That works well.

  13. nybgrus says:

    I’ve stopped calling it common sense and instead call it basic sense. It is basic, but it is not common, sadly.

  14. norrisL says:


    Being a lover of maths (how geeky is that?) I love your equation for its simplicity and accuracy; well done!

    Here in Australia, we ( Australian Veterinary Association) tend to use the term EBM as you would use SBM. In fact a recent AVA annual conference was titled “Evidence Based Medicine”. But I guess that what we SBM type vets are doing is first looking for prior plausibility before looking at any results. Sadly there are those among us who don’t look for prior plausibility and who are quite happy to accept an article printed in Dog Lovers Monthly as strong evidence for the use of quackery.

    If you want to see the type of garbage that one “vet” in Geelong practises, have a look at this site have a bucket handy in case it makes you sick

    The AVA has recently banned people like the Geelong vet from receiving continuing veterinary education points for subjects like homeopathy and other such favourites. These points are required in order to maintain your right to practice. If I was in charge of the AVA, the plan would be……take down your “vet” sign and hang up your “quack” sign so that your victims know who you really are.

    Have a great day Harriet

  15. Sastra says:

    In the absence of any other information, if I saw a specific reference to “evidence-based medicine” on a doctor’s website, business card, or office wall, I’d assume that he or she was trying to signal that they were not a big fan of alternative medicine.

    Would that be a reasonable way to bet? As far as I know, alt med supporters haven’t been notably trying to co-opt the term to signal something like “we may not have the science, but we have the evidence via personal experience to base this on wink*wink*.”

  16. Harriet Hall says:


    I fear that is not a completely reliable guide. I have a copy of the book “Somatovisceral Aspects of Chiropractic: An Evidence-Based Approach.” I perused it carefully but was unable to find anything in it that I could call evidence.

  17. BillyJoe says:

    I’m obviously missing something here.
    For me, the equation has always been: SBM = EBM + PP

  18. jt512 says:

    “The impact factor of 53.4% of the journals published was 0.”

    I wonder if they had as much fun writing that sentence as I had reading it.


  19. Harriet Hall says:

    For me, CT covers PP but includes other things, including thinking about possible biases and all the other things that can go wrong in research.

  20. Badly Shaved Monkey says:

    It’s been said before, but I’ll say it again. EBM, in it’s narrow technical sense, makes perfect sense when the underlying science does not flatly contradict the claims of a specific therapy. SBM agrees that basic science is neither sufficient nor necessary to sustain a medical claim: it is not enough to have results just from cells in culture, but we can also proceed where there is an absence of basic science. But, SBM insists, whereas EBM does not, that basic science must not absolutely conflict with the claim.

    Sufficient prior plausibility and extraordinary claims require extraordinary evidence are different ways of saying the same thing.

    I think SBM is trying to fill a blind spot in EBM that was not recognised when the project was instituted.

    I think Brennen’s comments about how the ‘higher’ levels of the hierarchy of EBM evidence can be distorted are really important. SCAMsters have seemingly deliberately constructed an edifice to provide support for their therapies that stands in parallel to EBM and looks quite like it, but is actually mere stage-scenery built without proper foundations.

  21. Janet says:

    I’m not sure how much of that was directed at my question, but thanks to everyone anyway. I think I will ask the doctor in question to clarify his usage of the EBM term as I’m not sure the finer meaning understood here is universally known or accepted. I know I have seen the term EBM used in brochures and other materials at clinics where it seems to imply a level of rigor than may not include SBM–not sure how one can know without detailed questioning in each case. I guess if the institution in question is “integrating” any woo at all, then you would have the answer in that case.

  22. James Coyne says:

    Quite relevant to this discussion. All Chinese RCTs evaluating acupuncture were positive.

    Control Clin Trials. 1998 Apr;19(2):159-66.
    Do certain countries produce only positive results? A systematic review of controlled trials.
    Vickers A, Goyal N, Harland R, Rees R.

    Research Council for Complementary Medicine, London, UK.

    To determine whether clinical trials originating in certain countries always have positive results.

    Abstracts of trials from Medline (January 1966-June 1995).

    Two separate studies were conducted. The first included trials in which the clinical outcome of a group of subjects receiving acupuncture was compared to that of a group receiving placebo, no treatment, or a nonacupuncture intervention. In the second study, randomized or controlled trials of interventions other than acupuncture that were published in China, Japan, Russia/USSR, or Taiwan were compared to those published in England.

    Blinded reviewers determined inclusion and outcome and separately classified each trial by country of origin.

    In the study of acupuncture trials, 252 of 1085 abstracts met the inclusion criteria. Research conducted in certain countries was uniformly favorable to acupuncture; all trials originating in China, Japan, Hong Kong, and Taiwan were positive, as were 10 out of 11 of those published in Russia/USSR. In studies that examined interventions other than acupuncture, 405 of 1100 abstracts met the inclusion criteria. Of trials published in England, 75% gave the test treatment as superior to control. The results for China, Japan, Russia/USSR, and Taiwan were 99%, 89%, 97%, and 95%, respectively. No trial published in China or Russia/USSR found a test treatment to be ineffective.

    Some countries publish unusually high proportions of positive results. Publication bias is a possible explanation. Researchers undertaking systematic reviews should consider carefully how to manage data from these countries.

  23. geo says:

    Also, certain groups of researchers who tend to publish together can produce more impressive results than other groups. It can be difficult to know how these results should be interpreted.

    I think it would be helpful to try to have some understanding as to what sort of impact we can expect in an RCT of a sham treatment run by researchers who believe in the efficacy of this treatment. Particularly for questionnaire scores like EQ-5D for QALY, which have a big impact upon assessments of cost effectiveness. Unless there are RCTs for a treatment showing a greater impact than that expected for sham treatments, maybe it would be a fair to assume that the treatment was not worth providing?

  24. Quill says:

    This has been a very timely post for me and my best friend who has been dealing with chronic pain for many years. I’ve been helping him by driving him to doctor’s appointments and navigating the world of pain management doctors. His insurance offered him a choice between two places, A or B. A was a small warm, friendly feeling place that had just hired a “naturopathic doctor” to provide complimentary medicine alongside an acupuncturist. B was a larger place, a bit more brisk, not as cozy, and no one on staff but MDs and DOs with the former almost all being double-board certified.

    He chose B and this article had something to do with that, so thank you.

  25. mho says:

    This accupuncture story (er, study) is making the rounds through some ovarian cancer networks.

    “*These authors equally contributed to the study.

    Dana-Farber and Mass General too??

    Ursula A. Matulonis, MD,3,* Julie E. Dunn, PhD,2 Hang Lee, PhD,4 Anne Doherty-Gilman, MPH,1 Elizabeth Dean-Clower, MD, MPH,1 Annekathryn Goodman, MD,5 Roger B. Davis, ScD,6 Julie Buring, ScD,6 Peter Wayne, PhD,6 David S. Rosenthal, MD,1 and Richard T. Penson, MD5
    1Leonard P. Zakim Center for Integrative Therapies, Dana-Farber Cancer Institute, Boston, MA.
    2The New England School of Acupuncture, Newton, MA.
    3Department of Medical Oncology, Dana-Farber Cancer Institute, Boston, MA.
    4Biostatistics Center, Massachusetts General Hospital, Boston, MA.
    5Department of Gynecologic Oncology & Medicine, Massachusetts General Hospital, Boston, MA.
    6Osher Center for Integrative Medicine, Harvard Medical School and Brigham and Women’s Hospital, Boston, MA.

Comments are closed.