Articles

Pragmatic Studies – More Bait and Switch

The course of research into so-called alternative medicine (CAM) over the last 20 years has largely followed the same pattern. There was little research into many of the popular CAM modalities, but proponents supported them anyway. We don’t need science, they argued, because we have anecdotes, history, and intuition.

When media attention, which drove public attention, was increasingly paid to CAM then serious scientific research increased. A specific manifestation of this was the National Center for Complementary and Alternative Medicine (NCCAM). CAM proponents then argued that their modalities were legitimate because they were being studied (as if that’s enough). Just you wait until all the positive evidence comes rolling in showing how right we were all along.

But then the evidence started coming in negative. A review of the research funded by NCCAM, for example, found that 10 years and 2.5 billion dollars of research had found no proof for any CAM modality. They must be doing something wrong, Senator Harkin (the NCCAM’s major backer) complained. They engaged in a bit of the kettle defense – they argue that the evidence is positive (by cherry picking, usually preliminary evidence), but when it is pointed out to them that evidence is actually negative they argue that the studies were not done fairly. But then when they are allowed to have studies done their way, but still well-controlled, and they are still negative, they argue that “Western science cannot test my CAM modalities.”

But at the same time, they cannot shake the need for scientific evidence to continue to push their modalities into the mainstream. The “your science can’t test my woo” defense only goes so far. So they have put themselves into a pickle – they demanded funding for CAM research, but now have to deal with the fact that the research is largely negative. CAM proponents are mostly not interested in finding what really works, and abandoning what does not work (I have an open challenge to anyone who can point to a CAM modality that was largely abandoned, rejected, or condemned by CAM proponents because of evidence of lack of efficacy). They are interested in using science to support and promote what they already believe works (a cardinal feature of pseudoscience).

And so they have entered the next phase of CAM research – the post-RCT (randomized controlled trial) phase. They have discovered the “pragmatic” study. You have to give it to them for their cleverness. A pragmatic study is meant to be a comparison of treatment options in real-world conditions. It studies treatments that already have proven efficacy from RCTs to see how they work, and how they compare, when applied in the less controlled environment of real-world practice. Pragmatic studies are useful for addressing the weaknesses of RCTs, mainly their somewhat contrived conditions (having strict inclusion criteria, for example).

But pragmatic studies are not efficacy trials themselves. They cannot be used to determine if a treatment works, because they do not control for variables and they are not blinded, so they are susceptible to placebo and non-specific effects. It is an abuse of the pragmatic study design to test a treatment that is not proven or to make efficacy claims based upon them. That has not stopped CAM proponents from doing just that.

In the news recently is just the latest example. Let me tell you the results of the study before I tell you what the study is of – just look at the data (I replaced the name of the treatment in the tables with just the word “treatment”):

This is what you need to know about the trial. There were two groups, treatment and control. Subjects were those with frequent doctor visits for symptoms that have not been diagnosed. While they were randomized, they were completely unblinded – everyone involved with the subjects knew who was getting treated and who wasn’t. At 26 weeks the control group was crossed over to receive treatment – so the difference up to 26 weeks is most important.

On two of the measures, quality of life and general practitioner consultation rates, there was no difference. In the other two measures there was a slight improvement in the treatment group, barely statistically significant. If this were an efficacy trial, this data would be unconvincing. What can fairly be concluded from this trial is that the treatment has no to minimal effect, and the tiny and inconsistent effect seen cannot be separated from placebo or non-specific effects. Further, it shows how anemic even placebo effects are for this treatment in this patient population. As a pragmatic trial, it’s essentially negative. As an efficacy trial, it’s worthless.

The study author’s, however, concluded that their treatment was effective and recommended it be incorporated into general practice.

The treatment in question is five-element acupuncture – acupuncture designed to balance the five elements of fire, water, metal, earth, and wood. This is the equivalent of balancing the four humors – in other words, it is pre-scientific superstitious nonsense. The study authors are all proponents. What is most amazing is how they got their paper, complete with unsupported conclusions, past peer-review.

The study authors are using the “part of this nutritious breakfast” con. Even as a child I understood that the sugary cereal, or whatever the commercial was trying to sell to me, was only “part of this nutritious breakfast” because the rest of the breakfast was nutritious all by itself. A doughnut would be “part of this nutritious breakfast.” So the authors of this study include five-point acupuncture with an hour of kind attention from a practitioner along with lifestyle advice and encouragement. They admit that they cannot separate out the variables here. Then they conclude that acupuncture is “part of this healthy regimen.” There is every reason to believe that it is an irrelevant part – as irrelevant as the doughnut is to the nutrition of a complete breakfast.

Conclusion

The pragmatic study bait and switch is here – it is now a firm part of the CAM strategy for promoting implausible therapies that don’t work. Editors and peer-reviewers need to be more aware of this scam so they don’t fall for it and inadvertently promote it, as they did in this case. Further, the results of this trial are not impressive even at face value – it’s basically negative, so it’s a double swindle.

All of this has not stopped the headlines from declaring that “acupuncture works” – which appears to be the only goal of such research.

Posted in: Clinical Trials, Science and Medicine

Leave a Comment (35) ↓

35 thoughts on “Pragmatic Studies – More Bait and Switch

  1. ConspicuousCarl says:

    [blockquote]And so they have entered the next phase of CAM research – the post-RCT (randomized controlled trial) phase. “[/blockquote]

    And my brain’s response was to replay a line from that SNL skit in which Tim Meadows is a census taker who is getting nothing but nonsense replies from Christopher Walken…

    “Let’s just proceed as if this is going really well.”

  2. “What is most amazing is how they got their paper, complete with unsupported conclusions, past peer-review”…

    It appears that the peer reviewed process has failed in this one instance. A true test of a research paper and it’s finding is whether it can be reproduced by another group of researchers. If it can not, then it’s findings and conclusions are usually not referenced and falls into obscurity.

    Dr Sam Girgis
    http://drsamgirgis.com

  3. S.C. former shruggie says:

    A true test of a research paper and it’s finding is whether it can be reproduced by another group of researchers.

    Weak to non-existent results from an unblinded pragmatic study of accupuncture? Oh yes, I’m confident others can reproduce those results. They may even exceed them, in being even less rigorous in design and even less statistically significant.

  4. synoeca says:

    this is all a bit sinister. can someone point me to the review of NCCAM alluded to above (10 years and 2.4 billion dollars of research) that found no support for CAM? thanks!

  5. I added the link: http://www.msnbc.msn.com/id/31190909/
    (and actually it was 2.5 billion, which I also corrected)

  6. daijiyobu says:

    The study design seems one step, one very small step, above the quality of evidence that results from use of a Ouija board…

    ‘More studies are needed’ [aka I've a few more years till I retire and I've run out of rigor].

    -r.c.

  7. pmoran says:

    Is there a reference to this study. Stephen?

    Further, it shows how anemic even placebo effects are for this treatment in this patient population. As a pragmatic trial, it’s essentially negative. As an efficacy trial, it’s worthless.

    The study’s value may lie in demonstrating to those who might think otherwise that the additional treatment is not, overall, a very cost-effective intervention in this clinical context.

    The study does not however preclude significant benefits for some included individuals, who may for some reason be more liable to the non-specific responses to this kind of medical interaction.

    It is also an outlier of a study. Most non-blinded controlled trials of acupuncture are more strongly suggestive of benefits from the added medical interaction. So the interesting question is “why not so much with this one?” Is it because this group of patients (undiagnosed conditions) are resistant to ANY kind of medical intervention? Or, is there something in the informed consenting of the subjects that is dampening non-specific responses?

    The pragmatic study bait and switch is here – it is now a firm part of the CAM strategy for promoting implausible therapies that don’t work

    The proponents of these methods believe they work. Their patients are telling them so all the time. We don’t need to imply sinister intent.

    I believe Peter Fisher was genuinely puzzled when he said, after yet another placebo-controlled trial of homeopathy showed no benefit: “Over these years we have come to believe that conventional RCTs [randomised controlled trials] are unlikely to capture the possible benefits of homeopathy . . . . It seems more important to define if homeopathists can genuinely control patients’ symptoms and less relevant to have concerns about whether this is due to a ‘genuine’ effect or to influencing the placebo response”.

    Our argument against the use of homeopathy and acupuncture is not based upon any certainty that they cannot contribute to patient well-being.

    It is a largely ethical stance that we have chosen to impose on ourselves, for mostly sound reasons, now that we have eaten of the tree of knowledge in a way that others have yet to do, and may never do.

    Also, if, as I strongly believe these methods are in practice simply recruiting all the non-specific elements of medical practice, then CAM practitioners have a point — the RCT as usually designed and applied will tend to inhibit “treatment” effects.

  8. moderation says:

    pmoran: “Our argument against the use of homeopathy and acupuncture is not based upon any certainty that they cannot contribute to patient well-being. It is a largely ethical stance that we have chosen to impose on ourselves, for mostly sound reasons …”

    I assume by this you mean the ethical stance concerning traetment with a known placebo. The problem with your point is that as you note: “… proponents of these methods believe they work.”. In fact these proponents have a religious-like belief in their treatments and exaggerate their effectiveness beyond the placebo effect, to the point that they promote them over truly efficacious treatment regimens.

    While you are correct that many proponents may not have “sinister intent” (some certainly do and are just snake oil salesmen) the results of their behavior is the same as if they did have “sinister intent”(encouraging patients away from effective therapies). If I kill a jogger with my car does it matter to the jogger whether I did it intentionally or I was distracted on my call phone? Only one of those is “sinister”, but the jogger is dead either way.

  9. Mark Crislip says:

    My real world pragmatic, non SBM rule of thumb is that in the real world, diagnostic and theraputic interventions are about half as good as they are in clinical trials.

    So CAMS must really stink on ice.

  10. Anthro says:

    Whenever I get into a discussion of this nature with a woo-person, I often hear that he or she does not care if it is “only a placebo effect” as long as he or she “feels better” after the treatment.

    At this point my response is, “perhaps, but why should you hand over $50/hr (or whatever) for a placebo to someone who is claiming that it is the woo, when it is just the placebo effect, unless you have more money than sense?”

    This sometimes puts them of a mind to actually listen long enough for me to refer them to this blog, and plants just enough of a seed of doubt that they actually visit. I’ve had two successes recently after a long period of despair that I could reach these people in any way. But maybe it’s just a coincidence.

  11. Ken Hamer says:

    What’s wrong with the following phrases:

    - “10 years and 2.5 billion dollars of research had found no proof for any CAM modality”
    - “What can fairly be concluded from this trial is that the treatment has no to minimal effect”
    - “the results of this trial are not impressive even at face value”?

    In a word, they are too “polite.” They are filled with weaselly language that almost satarizes the nonsense it is intended to critique.

    I think it’s long past the time where the pro-science/evidence community became as rabid and vicious as the clowns who attack Dr Offit and others like him. The thing is, you don’t even need to stoop to the kinds of hyperbole and outright lies they use. You simply need to use plain-talking, straight-shooting, everyday language, that everyone can understand. Language that puts sCAM under the light it deserves.

    I think all it takes is more frequent use of a single word: “failure.” It needs to be made clear just how frequently and how reliably sCAM nonsense fails. A simple rephrasing is all that is required.

    For example, here are my “translations” of the phrases I noted above:

    “10 years and 2.5 billion dollars of research had found no proof for any CAM modality”

    becomes

    “10 years and 2.5 billion dollars of research failed to find any proof for any CAM modality”

    “What can fairly be concluded from this trial is that the treatment has no to minimal effect”

    becomes

    “What can fairly be concluded from this trial is that the treatment has failed completely or failed almost completely”

    “the results of this trial are not impressive even at face value”

    becomes

    “the results of this trial are essentially complete failure even at face value”

    Put simply, every time the word CAM or accupuncture or anything else along that line is used, then the word “failure” should also be used in conjunction with it, strongly and repeatedly, everytime. People need to understand that it’s not a matter of the proof hasn’t yet been found, but rather one failure after another, as certain as the rising sun.

    It’s long since time that sCAM became understood as the colossal failure it is.

  12. Prometheus says:

    When I read:

    “A review of the research funded by NCCAM, for example, found that 10 years and 2.5 billion dollars of research had found no proof for any CAM modality.”

    …in juxtaposition with:

    “But then when they are allowed to have studies done their way, but still well-controlled, and they are still negative, they argue that ‘Western science cannot test my CAM modalities.’ “

    …I had a bit of an epiphany. If “Western science” can’t test “CAM”, then why are the US taxpayers spending 2.5 billion dollars over ten years to fund scientific (presumably “Western” science) studies of “CAM”?

    Isn’t that rather like saying “You can’t photograph ghosts with a camera.” and then spending millions of dollars setting up cameras to get ghost photos?

    It seems to me that the justification for the NCCAM just vanished.

    Prometheus

  13. WilliamLawrenceUtridge says:

    @Prometheus

    Not really; your rational holds if we actually listened to the CAMsters. However, there is some small merit in actual scientific study of CAM modalities for possible (but unlikely) health benefits, and if nothing else then to expand our understanding of nonspecific effects like demand characteristics, bedside manner and different ways of enhancing the placebo effect. So long as we agree to use science and listen to what it says, there’s use in the research.

    Of course, one can argue (as probably most people who blog here have already) that the NCCAM is unnecessary because it throws good money after bad, it funds sloppy research that could be better conducted by other agencies, does research on modalities that are highly unlikely and would be better tested in bench research, etc. The real problem is seeking the prestige of science without the rigor, or the willingness to acknowledge that your modality has been disproven and should be abandoned.

  14. Scott says:

    The real problem is seeking the prestige of science without the rigor, or the willingness to acknowledge that your modality has been disproven and should be abandoned.

    The latter, I think, is the deeper issue. Initially, they’re all for scientific testing of their pet woo, and the more rigorous the better. After all, they already KNOW what the answer is going to be, so the more compelling that result is the better.

    Where they get into trouble is when the science doesn’t give the answer that they KNOW is correct. But they KNOW that it’s correct, so the only explanation is that the science was wrong. Then they have to rationalize away the negative result by saying “well, I guess we learned more about the limitations of science.”

    Each position is consistent with what they were feeling at the time. But the ultimate result is that CAM is claimed to be scientifically testable precisely to the extent that those tests give the results that the wooists KNOW are right. Results-driven acceptance.

  15. suewv says:

    Let’s do a study on how you rational right-brainers actually feel insecure in your Science, resorting to derogatory terms such as “woo” and “they” to refer to those who are more open-minded to centuries-old methods of treatment. What are you afraid of? Look deep inside ~ I know that might be hard to do…

  16. suewv says:

    Correction: rational LEFT-brainers.

  17. Scott says:

    Tell me, sue, are you also “open-minded” to the possibility that the Earth is flat? It’s pretty much the same thing.

  18. Prometheus says:

    @WL Utridge;

    I seems that the only purpose of the NCCAM remaining is to find out if any of the “CAM” modalities actually work so that they can be subsumed into “real” medicine. After all, the “CAM” proponents have pretty much said that they will not accept any scientific results that fail to vindicate their a priori belief that “CAM” is safe and effective.

    Given that, it seems wasteful for the NCCAM to research “CAM” modalities that have a low prior probability of efficacy – e.g. homeopathy, acupuncture and any of the “energy field” therapies. Since we’ll never convince the “true believers” that their dogma is wrong, why waste time and money trying?

    There may be some “nuggets” of real medicine in the manure pile of “CAM”, but we won’t find them by repetitively demonstrating that homeopathy and acupuncture don’t work.

    Prometheus

  19. nybgrus says:

    Actually sue, the reason we call it “woo” is because we are secure in our science and it tells us it is woo.

    Just because something is old, doesn’t make it true, no matter how much you wish it to be. Blood letting is very old and has a very long tradition of use. Why don’t you advocate for that?

    Keep an open mind, but not so open your brains fall out.

  20. suewv says:

    Leeches and electroshock have both come in and out of fashion, have they not?

  21. Harriet Hall says:

    @suewv, “Leeches and electroshock have both come in and out of fashion, have they not?”

    Not exactly. They went “out of fashion” for earlier indications that were proven wrong. They are now “in fashion” for specific indications that have been proven effective by science.

  22. nybgrus says:

    indeed. and leeches and bloodletting are very different things. The only thing close to bloodletting these days is for hemachromatosis.

    leeches are used exclusively (unless someone else here knows another application I am unaware of) for re-attachment surgeries to prevent swelling and blood stasis.

    Bloodletting was literally bleeding you of a litre or so (sometime more!) of blood to try and cure pretty much all that ails you. It was thought to “work” for hundreds (if not thousands) of years. Then that pesky science showed it really didn’t do anything except harm.

    So once again, why don’t you advocate for bloodletting as one of the most ancient, long standing, and widely revered methods of healing on record?

  23. suewv says:

    Old isn’t my only criteria.

  24. nybgrus says:

    then can you please define what your criteria is? and if old is just a part of that criteria can you explain how and why that factors into a decision of safety and efficacy?

  25. JPZ says:

    Mark Crislip said: “My real world pragmatic, non SBM rule of thumb is that in the real world, diagnostic and theraputic interventions are about half as good as they are in clinical trials.

    So CAMS must really stink on ice.”

    ROFL! That is so unscientific; but in my n=1, outlier, placebo-effect, self-delusional experience; also so dang true! But, at least I admit that I have no scientific basis for this observation.

    *wipes tear from eye* Dang good laugh. :)

  26. WilliamLawrenceUtridge says:

    Sue, in science the only criteria that matters is evidence. The germ theory of disease could be discarded given sufficient good-quality evidence, it’s just really unlikely. If your criteria is anything but “evidence” (with an asterisk next to it specifying “good evidence” rather than anecdote) then you’re not doing science, you don’t understand science, and you’re missing the point of this blog. Scientists and skeptics don’t hate CAM, they just understand enough about the body to realize it’s extremely unlikely that most of it is effective.

  27. Scott says:

    Not just understanding of the body, either. In most cases there’s actual concrete evidence demonstrating that it’s ineffective.

    Really, it’s sue and her ilk who are close-minded. Any fact that doesn’t conform to their preconceptions is summarily ignored.

  28. sue 07
    “Correction: rational LEFT-brainers.”

    As a “right-brainer”, I resent the implication that I’m irrational. I also want to know when the word “they” became derogatory.

    And, oh yeah, the “left brain/right brain” thing is highly exaggerated.

    http://www.spring.org.uk/2008/03/two-brains-for-price-of-one.php

  29. suewv says:

    I threw myself into the pack of wolves ~ now I’m pulling myself out.

  30. weing says:

    “I threw myself into the pack of wolves ~ now I’m pulling myself out.”

    You mean critical thinking is too much for you? Go back to sleep.

  31. nybgrus says:

    Sue – we are asking simple and reasonable questions. For us “it just is” or “cuz I like it” are not acceptable forms of argument or rationale.

    The fact that you cannot answer my simple questions means you don’t have the answers and should reconsider your stance. You even said to us:

    What are you afraid of? Look deep inside ~ I know that might be hard to do…

    I very politely asked you just a few questions – one that asked you to look deep inside and provide some answers. Looks like that might have been too hard for you to do.

    What are you afraid of?

  32. I’m sure that sue did not mean to be derogatory in referring to posters as a “pack of wolves”. Everyone knows that wolves are an integral part of an ecosystem. They are also loving parents and exist peacefully with their family unit in the wild.

  33. nybgrus says:

    @michelle: I do always love your posts. :-)

  34. @nybgrus
    thanks
    &
    ditto. ;)

Comments are closed.