Articles

Acupuncture for Hot Flashes – Or, Why So Many Worthless Acupuncture Studies?

In yet another round of science by press release, a particularly unimpressive acupuncture study is making the rounds of the major news outlets proclaiming that acupuncture works. I guess that is a sort-of answer to my title question – why are so many scientifically worthless acupuncture studies being done?

Let’s take a look at this particular study to see why it is so weak. All I have to go on is the press release, since the study is not published. It was presented at a scientific meeting – which is legitimate, I just don’t have access to it. (The bar for publication in a peer-reviewed journal is much higher than presentation at a meeting, and there may, in fact, be changes to the text prior to publication.) But we can still say a great deal about this study from the information provided.

Acupuncture for Hot Flashes

The study looks at acupuncture vs Effexor for the treatment of hot flashes and other symptoms in women who have had breast cancer and are on Tamoxifen or Arimidex. These therapies cause vasomotor side effect, like the hot flashes or night sweats. Normally hormonal therapy can be used to treat these symptoms, but women who have had certain kinds of breast cancer cannot take those hormones so other treatments are necessary. One standard treatment is the anti-depressant Effexor (venlafaxine – a serotonin/norepinephrine reuptake inhibitor).

There were 47 patients in the study followed for 12 weeks in one of two arms – either acupuncture or standard treatment with Effexor.  The study showed that acupuncture was as effective as Effexor but had fewer side effects and lasted longer after treatments stopped.

The press release also reports:

Findings also show there were additional benefits to acupuncture treatment for breast cancer patients, such as an increased sense of well being, more energy, and in some cases, a higher sex drive, that were not experienced in those patients who underwent drug treatment for their hot flashes.

Evaluating Clinical Trials

Readers of this blog will know that not all clinical trials are created equal, and there are some basic assessments that should be made in order to properly weigh the results of a study.

This study has many weaknesses – enough to make the results uninterpretable, which is jargon for “worthless.”

The fatal problem with this study is that it is not blinded – there is no sham acupuncture. Patients, therefore, knew if they received the novel treatment that was being studied. The long list of non-specific “benefits” perceived by the subjects who received acupuncture (such as an increased sense of well-being) is evidence for the placebo effect in this group.  This weakness is enough to condemn the study.

Further, the study is small, with only 47 subjects. This is especially a problem for an equivalence outcome – the study showed no difference in degree of benefit between acupuncture and Effexor (although there was a difference in duration). This could simply mean that the study was not powerful enough to see any difference.

This small study size is also exacerbated by the fact that there was no placebo group – acupuncture was being compared to Effexor. It is therefore possible that in this study neither treatment worked – since we do not have a placebo control to gauge the absolute effectiveness of either treatment.

Prior studies indicate that Effexor is only “moderately” effective in treating hot flashes. So it is not an unreasonable control treatment, but the modest effectiveness of Effexor as a treatment makes the small sample size even more problematic.

Also, Effexor does have side effects because it is biologically active. It may be that Effexor has moderate benefit for hot flashes but these benefits are offset by the side effects in some cases. Meanwhile, acupuncture may have no effects or side effects -and so the net effect may have been similar.

The Disconnect Between Science and Press Release

Such a study as this is best described as a pilot study – meaning a small preliminary study. I have nothing against pilot studies – they serve an essential purpose in the arc of clinical research. They are useful for deciding if a clinical approach is worthy of further more definitive study. They should not be used as a basis for clinical decisions, however. They certainly should not be used to conclude that a highly implausible treatment has biological function.

There are two problems with this study, however. The first is the press release, which states:

“Our study shows that physicians and patients have an additional therapy for something that affects the majority of breast cancer survivors and actually has benefits, as opposed to more side effects. The effect is more durable than a drug commonly used to treat these vasomotor symptoms and, ultimately, is more cost-effective for insurance companies,” Eleanor Walker, M.D., lead author of the study and a radiation oncologist at the Henry Ford Hospital Department of Radiation Oncology in Detroit, said.

I find these statements to be completely misleading (to the point of being irresponsible) and not justified in any way by this study. It would only have been appropriate to say that this study shows acupuncture “may” be of use but requires further study – that’s what pilot studies are for.

The media, however, picked up the press release without criticism or meaningful analysis.

The Arc of Clinical Trials

My second problem with this study is to question the purpose of doing pilot studies at this stage of acupuncture research. Clinical research has a typical progress. It begins with observations, usually published as case reports or case series, suggesting a possible new connection or treatment. This then progresses to pilot studies (like this one) which are small and may be blinded or open.

If pilot studies are promising, or at least rule out any major risk or downside to a treatment, then larger studies are done that are designed to be definitive. However, this usually takes a few rounds, with protocols being refined in response to criticism or alternate interpretations of the data. Eventually, truly definitive or consensus trials are done and the clinical question is largely settled. (New data, however, can always revive an old question.)

Acupuncture has been studied for years, and trial design has evolved to well-designed trials that are double-blind and controlled by either sham acupuncture (where needles are inserted, but in the “wrong” place or not to depth) or, even better, placebo acupuncture where opaque sheaths hide from the practitioner and the patient whether or not a needle is inserted at all (the pressure from the sheath can disguise the sensation of the thin acupuncture needle being inserted).

At this point there is no reason to do pilot studies for acupuncture – this represents a regression of the typical research process. The only research that is of value at this point are well-blinded trials. However, such trials have failed to demonstrate that acupuncture has any specific effect (anything beyond placebo effects or perhaps non-specific effects from the process of getting acupuncture, such as relaxation).

It is reasonable to speculate that proponents of acupuncture are shying away from well-controlled trials because they have not given the positive results they desire. If this is not true, and pilot studies are being performed to explore the potential of specific applications of acupuncture, then the statements of these researchers in their press releases and to the public should reflect that – but they don’t. Rather, poorly designed or preliminary studies (which at this point are scientifically worthless) are used to make glowing endorsements of the effectiveness of acupuncture to the public.

This is not an isolated case. Three weeks ago a similarly worthless study of acupuncture for the treatment of headaches was published. This study, too, was not blinded and yet was used to promote the alleged effectiveness of acupuncture.

Conclusion

Acupuncture remains a highly implausible treatment. For this reason the threshold for evidence should be higher than for more plausible treatments. Acupuncture has also been studied enough for the technology of acupuncture research to have matured to fairly definitive studies, and to conclude that there is probably no large or easily detectable biological effect from acupuncture.

In addition, the popularity of acupuncture (while still small in absolute numbers) far outstrips the evidence for its effectiveness, making acupuncture a controversial treatment. This controversy is exacerbated by the existence of dedicated practitioners (acupuncturists), who have a vested interest in this one modality.

For these reasons any further testing of acupuncture should aspire to the highest scientific standards. Pilot studies of acupuncture are worse than worthless – they do nothing to further the scientific question, and they are abused to promote a dubious treatment through the credulous media.

Posted in: Acupuncture, Clinical Trials, Energy Medicine

Leave a Comment (20) ↓

20 thoughts on “Acupuncture for Hot Flashes – Or, Why So Many Worthless Acupuncture Studies?

  1. zayzayem says:

    Wow! That press release, “Acupuncture reduces side effects of breast cancer treatment as much as conventional drug therapy”, is even more misleading than the one I sent into this website.

    This tested against a second-rate treatment for hot-flashes, a post-menopausal condition.

    It did nothing at all to the patients breast cancer at all.

  2. vinny says:

    Very nice review of medical study methods and problems with accupuncture. I wish this review had as wide a distribution as the press release.

    Steven,
    The following is an interesting venture into the realm of parapsychology by a very respected medical center. I am curious about your thoughts on this subject.

    http://news.yahoo.com/story//time/20080923/hl_time/whathappenswhenwedie

  3. vinny says:

    I should be more specific in my question: Not your thoughts on parapsychology but why would Cornell Medical Center engage in such activity and whether you know anything about Dr. Sam Parnia .

  4. mba says:

    As a first year medical student, I am constantly surprised by the credulity with which acupuncture is mentioned in some lectures and history & physical group trainings. Probably this study will only increase the problem. Acupuncture seems to be the favorite woo of medical schools, although I sometimes hear people speaking of homeopathy with a straight face. During one standardized patient session, it was suggested by an FP resident that we could suggest acupuncture for back pain complaints if the patient was open to it. Wow. And here I thought part of the doctor’s job is to help the patient find EFFECTIVE treatments that won’t waste his time and money. But no, you can suggest acupuncture if he likes it. I had to bit my tongue before I blurted something out that would probably get me in trouble, something like, “you can try acupuncture, as long as you are aware that it is for ENTERTAINMENT PURPOSES ONLY!”

  5. …although I sometimes hear people speaking of homeopathy with a straight face.

    Now that‘s depressing, given that homeopathy is about as obviously quackery as it is possible to be.

    By the way…

    http://scienceblogs.com/insolence/2008/09/yawn_yet_another_worthless_acupuncture_s.php

  6. Karl Withakay says:

    It would be interesting to do an intentionally unblinded study on the power of placebo and suggestion.

    Take two groups:

    Give one acupuncture (or sham acupuncture) and tell them it is normal acupuncture.
    Give the other group the same, but tell them you are testing out newly discovered, more powerful Chi points (perhaps from some newly discovered ancient Chinese manuscripts) that are believed to have a much more powerful effect, but also produce strong side effects (nausea, dry mouth, headache, difficulty sleeping, constipation, fatigue, etc)

    and note the differences in reported side effects.

  7. Harriet Hall says:

    In Bausell’s book Snake Oil Science he describes acupuncture research he was involved in at NIH. A good sham control was compared to “real” acupuncture. The results were equivalent, but the neat thing was that they asked patients afterwards which group they thought they were in. If they thought they got the real thing, it was more likely to work – no matter which group they were actually in!

  8. mba says:

    So, my question is, how does one control for patient bias? With something like acupuncture, most patients have probably heard about it, and have favorable opinions already, or else they would not have enrolled in the study. Without blinding, already predisposed patients will just report what they already thought about acupuncture. And where do they find these patients? What’s to stop an acupuncture lobbying group from recruiting patients into a trial, and how does a legitimate trial control for patients who may have an agenda to produce a favorable outcome, or an outcome that can be spun favorably for the news media? Even with blinding, such problems could cloud the outcome of a trial, since you might have patients who are likely to try to discover which group they are in, and report accordingly. Since acupuncture is difficult to blind anyway (needles must penetrate skin to be “real” acupuncture), this again brings up the problem of patient advocates infiltrating the trial. It seems that often, when conducting clinical trials, it is just assumed that the patients themselves are not the source of any bias.

  9. urology-resident says:

    My wife is 7 weeks pregnant and having severe nausea and vomiting. I’ve read every article and book discussing nausea and vomiting of pregnancy and I was very dissapointed at seeing every one of them recommending accupressure and accupuncture. Some of them admit that the evidence is weak but seems that most of them recommed them at least for the placebo effect and because there are no side effects. This was all the way from review articles, to OB-Gyn textbooks etc…

  10. NPMommy says:

    Urology-resident- I almost hate to admit it on this site but I used an accupressure band when I was pregnant for my nausea. My OB didn’t recommend it to me- I read about them in a book or the internet I can’t remember. I thought- it probably won’t work, its probably just placebo but I was miserable. I found one for only $11 and I decided to give it a chance- especially since there was no risk to my baby. And it worked. And it probably was placebo but frankly at that point I didn’t care because I felt better.
    I know your point is more that they are in OB/GYN textbooks and the lack of evidence for them but when you are pregnant and miserable I can understand trying almost anything (as long as it is safe) to help you feel better. (and I can kind of understand recommending them as well since there isn’t too much to offer patients anyway).

  11. pmoran says:

    NPmommy, to all appearances you have found a perfectly safe and extremely cheap way of overcoming a distressing problem. Even better, you are not over- interpreting what happened. You understand that your anecdote is not strong evidence for old Chinese medical beliefs Neither is the study that Steve criticizes, of course.

    But you do highlight an awkward matter, for medical skeptics –the unmentionable elephant that everyone squeezes around in the dining room. What are the practical medical implications of considerable evidence that relatively safe, cheap, non-drug measures like acupressure can help patients with what can be otherwise difficult and distressing conditions?

    Even the authors of this acupuncture study are not putting it forward as supporting traditional acupuncture theory. One is quoted thus: “A substantial placebo effect found in other studies of hot flashes suggests that attention alone or expectancies on the part of study participants may be sufficient to reduce hot flashes,” ( http://www.medscape.com/viewarticle/581109?sssdmh=dm1.389394&src=nldne )

    There is little question that treatment oddities like acupuncture can direct the user’s mind away from symptoms, or trigger beneficial re-evaluation of them, if appropriate expectations are aroused.

    When sailing with a doctor friend I began to feel quite sea-sick. He immediately gave me the tiller, directing me to keep the boat on a certain heading into the wind. Concentrating on something I had never done before produced instant and complete relief of sea sickness. Sometimes sickness is a matter of where attention is directed.

    A ridiculous example of inert treatment in action: when the sea started to get a little rough on a boat tour out of Strahan, Tasmania, the crew immediately began circulating with pieces of ordinary cotton wool. Anyone feeling queezy was told to (“quickly!”) put a wisp of it in the LEFT ear. “It works!” the crew said emphatically. I am confident it does, for many, and that it would also be shown to do so in most comparisons with a “no treatment” arm in a RCT. What a clever way for the crew to reduce the messes they have to clean up without getting into the complications of mass medicating the public with drugs, which may take too long to act, anyway!. Almost certainly once one person started to vomit others would follow like dominoes.

    I think this is a brilliant example of the use of a placebo/distractant.. Yes, I know a controlled trial comparing voyages with and without cotton wool is desirable, but do we not now have enough such non-placebo controlled trials to be able to predict their likely results? It is the very reason why we insist on a well-blinded placebo when the intrinsic efficacy of a treatment (as opposed to its placebo/distractant/counterirritant/reassurance/relaxation-inducing impacts) is being assessed. And we certainly don’t need a study to prove that cotton wool in the ear has no mysterious healing powers.

    I suppose there is some elegant ethical argument as to why a side-effect prone drug should nevertheless be our first choice of treatment for hot flushes that are themselves a drug side-effect, even if the results are the same in this preliminary controlled study (Unlike Steve, I am quite comfortable with the likelihood that a larger study would give much the same results ) . Or, is the concern that acupuncturists might get too uppity altogether, if we allow them to treat hot flashes?

  12. Harriet Hall says:

    There is a difference between saying you’ve seen symptoms resolve with a treatment and saying there is scientific evidence that it works better than placebo. You don’t have to give up your ethics.

  13. Ah, Peter,

    As usual, I agree with you almost completely. But when you write, “I suppose there is some elegant ethical argument as to why a side-effect prone drug should nevertheless be our first choice [etc.],” I feel that you are, once again, provoking “the unmentionable elephant” that you’ve mentioned, and I’ve responded to, several times. This time, however, I see an opening for common ground. You write:

    “And we certainly don’t need a study to prove that cotton wool in the ear has no mysterious healing powers.”

    Yes! I agree! Nor, it seems to me, do we need “a controlled trial comparing voyages with and without cotton wool” or any more studies of acupressure for nausea or of acupuncture for hot flashes, controlled or not. These kinds of things have been around for long enough, and we know enough about distraction and other rather obvious components of the “placebo effect” to simply accept them for what they are, as opined by NPMommy: “placebo but frankly at that point I didn’t care because I felt better.” Chicken soup without the potential for hypernatremia or hypovolemia.

    Now, what about other worthless studies that IMC pushers are so eager to keep doing? Some are even not so bad, in the strictly methodologic sense, but they’re all silly. They are a caricature of Evidence-Based Medicine, both because they lack any acknowledgment of prior probability but also for the reason that you, Peter, have been driving at: it really doesn’t matter what the outcomes are for the trivial claims (acupressure), because the wise physician ought to view them as though they are so many chicken soups. The non-trivial claims (Gonzo, chelation) ought not to be studied in humans without the usual preliminaries (laboratory and animal studies) showing promise. Otherwise there is no way to study them safely and ethically, which is precisely what we’ve seen.

    Just as it makes no sense to “study” most folk remedies (with occasional obvious exceptions, such as teasing a potentially useful molecule from some oral or inhaled concoction), it also makes no sense to grant formal credentials to those who profess expertise in their practice. This is different from your suggestion that we bloggers would probably agree with “some elegant ethical argument as to why a side-effect prone drug should nevertheless be our first choice…” I, for one, have no problem with physicians who offer chicken soup or distractions for complaints that are minor and self-limited (even if very unpleasant) when there isn’t a very good specific treatment. I just don’t think we should tell patients stories about “Qi” or “meridians” or “energies” or “like cures like.”

    On the other hand, I think that studies should be limited to rational, specific treatments, because sooner or later good ones will be found. Ondansetron was real improvement over previous anti-nausea agents for chemotherapy, and eventually there will probably be one or more drugs that really do, safely and effectively, abolish nausea. Eventually there’ll be a drug or drugs that safely abolish the symptoms of a URI.

    In the meantime, let’s keep folk remedies in the category of folk remedies and not attempt to make them seem equivalent to specific medicines, either by “studying” them or by legitimizing them as specialties. There IS a concern that “acupuncturists might get too uppity altogether, if we allow them to treat hot flashes,” if one grants that a license to practice acupuncture makes “allow” synonymous with “promote.” An example follows.

    In Massachusetts, the acupuncture board is under the auspices of the medical board, although the acup. board has a distinct membership consisting mostly of acupuncturists. Here are the words of a former chairman of the medical board, written a few years ago:

    “At today’s [acup. board] meeting, a woman lodged a complaint against her acupuncturist for exceeding his scope of practice. She saw him for weight gain, fatigue and positive antithyroid antibodies, and wanted to discuss rebalancing her diet and receiving acupuncture to restore her chemical balance. She received a full medical exam, including a breast exam in which he claimed to palpate a breast lump but was able to assure her that it was not malignant, even though her sister had come down with breast cancer 10 years ago. He took a full medical history, and reported to the committee that this was done in compliance with oriental and traditional medicine. Nowhere in the investigation of this complaint did the acupuncturists [on the board] even address the question of scope of practice.”

    That episode came to light only because of the patient’s complaint, so it is a safe bet that there are many more such episodes that no one hears about. What’s the answer? Stop the idiocy of pretending that such practices and practitioners can be regulated (and hence recommended) in any meaningful way, other than by invoking existing anti-fraud laws, assault laws, and laws prohibiting the illegitimate practice of medicine. Let both buyer and seller beware, and I’m willing to bet that sellers will become less uppity and each will become rarer and more careful, just as they were before the current wave of sCAM. (see: http://www.sciencebasedmedicine.org/?p=141 for more discussion)

    Are we converging, Peter?

  14. pmoran says:

    I only partially agree about the need for studies. Both our positions are tentative. You need better evidence than you have for your judgment that the potential benefits to patients from ritualistic or folk treatments are too small to be worth worrying about. This is what enables you to focus on peripheral concerns that are themselves a matter of opinion,

    I contend that we already have a fairly well-developed scientific rationale for the use of such treatments. This can be set against the superstition and pseudoscience. I also know that it would not be at all difficult to assemble a panel of trustworthy acupuncturists should the need arise. I am probably as galled as you by the inevitable misplaced triumphalism of traditional acupuncturists whenever this method seems to be more widely accepted, but I am prepared to wear that if it should turn out to be in the interests of optimal patient care.

    For my part, I need more evidence that any specific method can be reasonably cost-effective and durable in effect under the conditions that would apply to its use within or alongside present conventional medical practice. That need can justify certain types of unblinded clinical study similar to the present one. The measures used need to be as objective as humanly possible.

    Yes, drugs will certainly be better by the year 2100 but we have to treat patients within the present historical environment. I think the upsurge of “alternative” medicine is partly due to a premature retreat of doctors into an unnecessarily restrictive evidence-based shell, and overoptimism concerning the pace of medical progress. We have aroused expectations that bioactive methods cannot yet safely satisfy.

Comments are closed.