Articles

Our Visit with NCCAM

Over the past two plus years of the existence of Science-Based Medicine (SBM) we have been highly critical of the National Center for Complementary and Alternative Medicine (NCCAM) – going so far as to call for it to be abolished. We are collectively concerned that the NCCAM primarily serves as a means for promoting unscientific medicine, and any useful research it funds can be handled by other centers at the NIH.

So we were a bit surprised when the current director of the NCCAM, Josephine Briggs, contacted us directly and asked for a face-to-face meeting to discuss our concerns.

That meeting took place this past Friday, April 2nd. David Gorski, Kimball Atwood and I met with Dr. Briggs, Deputy Director Dr. John Killen, Karin Lohman PhD (Director, Office of Policy, Planning, and Evaluation) and Christy Thomsen (Director, Office of Communications and Public Liaison).

Dr. Briggs very graciously began the meeting by telling us that she and her staff have been reading SBM and they find our arguments to be cogent and serious. She shares many of our concerns, and feels that we are an important voice and are having an impact. She then essentially turned it over to us to discuss our primary concerns regarding the NCCAM.

We were prepared for this.

I first pointed out that many of our concerns deal with issues that are outside the purview of the NCCAM director (such as regulation) and therefore we would not bring them up but would rather stick to constructive feedback and concrete ways in which the NCCAM can better serve its mandate. These issues broke down as follows:

NCCAM1s

Left to right: David Gorski, John Killen, Steven Novella, Kimball Atwood, Josephine Briggs, Christy Thomsen, Karin Lohman

Ethical Concerns

The NCCAM faces particularly complex ethical issues in funding some clinical trials because of the very nature of the topics it is tasked to research – those with low plausibility that have been bypassed by mainstream research. The ethical guidelines of clinical research dictate that before subjecting a person to an experimental treatment, there is sufficient evidence for safety and plausibility of benefit from pre-clinical and animal studies. It might therefore be considered unethical to subject people to experimental treatments that are highly implausible.

The particular study that is most concerning is the TACT trial, and Kimball reviews the specific details of concern here. TACT is a trial of chelation therapy for heart disease. The concern is that chelation therapy is not a benign treatment and there already has been sufficient evidence to conclude that it does not work. Further study is therefore unethical.

Dr. Briggs acknowledged our concerns, but pointed out two things. First, this study came into being before her tenure at NCCAM (she became director on January 24th 2008). Second, the TACT trial has been turned over to NHLBI (National Heart Lung and Blood Institute), who now sponsors the trial, while the NCCAM still partly funds and collaborates on the trial.

What this means is that Dr. Briggs was able to decline to comment on the TACT trial on the grounds that it falls under the aegis of the NHLBI. This effectively cut off discussion on this topic, which is unfortunate.

This does bring up another issue – the NCCAM funds many studies along with other centers at the NIH, and (as with TACT) they intend to allow centers with the proper expertise to take the lead. NHLBI does heart studies, so they took over TACT. This is reasonable, but does have the consequences of effectively increasing the amount of research funding the NCCAM controls, and also provides cover (intended or not) for controversial studies like TACT.

Kimball intends to follow up with the NHLBI regarding TACT and will likely give us an update.

TACT aside, the ethical concerns remain and this is an issue we will have to follow with future studies.

Types of Studies funded by NCCAM

Another core issue we discussed is the fact that the NCCAM funds many studies that are designed to promote CAM in general or specific CAM modalities rather than study whether or not they are effective. Studying how CAM is used, or barriers to CAM acceptance – prior to demonstrating that any particular CAM modality actually works, is putting the cart before the horse.

But there is a more subtle and insidious problem. So-called pragmatic studies are trials that either compare different treatments or follow outcomes for one treatment in real-world practice. They are often not rigorously blinded nor are variables controlled. They are typically “intention to treat” trials where everyone is followed, regardless of whether or not they complied with the treatment.

Pragmatic studies are a very useful way of tracking real world outcomes. It may be true that aspirin reduces strokes and heart attacks, but what happens when a typical primary care doctor prescribes aspirin? Are their patients compliant? Do they run into side effects or other problems that cause them to stop taking the medication? What do primary docs have to do to improve compliance and minimize side effects? All good questions.

But such studies are simply not designed to answer the question – does aspirin work for the reduction of heart attacks and strokes. Efficacy trials are needed for that.

What we have observed in the CAM world, however, is that pragmatic trials are being performed on treatments that have no proven efficacy, and the outcomes are being misinterpreted and presented as evidence for efficacy. For some modalities, such as acupuncture, this is a very deliberate strategy and is being done in response to well-controlled efficacy trials that are negative.

We would therefore like to see NCCAM focus on efficacy trials, especially for treatments that do not already have proven efficacy. Pragmatic studies of unproven therapies are inappropriate and are ripe for abuse.

Dr. Briggs response on this issue was equivocal – she defended the utility of pragmatic studies but also acknowledged our concerns. We ran into the same problem in that, any examples of such behavior more than 2 years old were before Dr. Briggs time. So we will have to simply monitor things going forward.

Never Say Never

Related to the issue of what kinds of studies the NCCAM should fund is the following question – are there any treatment modalities that have been sufficiently shown to be both implausible and lacking in efficacy that the NCCAM should close the door on future research. When is enough enough?

We used our favorite example – homeopathy, which is especially pertinent following the report of the House of Commons Science and Technology Committee in the UK, who concluded that homeopathy is worthless, cannot possibly work, and should be abandoned in all ways.

It seems to us that the NCCAM (at least so far) has never closed the door on any modality, no matter how implausible and no matter how much evidence for lack of efficacy there is. This seems, if nothing else, like a waste of taxpayer money.

Dr. Briggs response was that in the last two years (under her directorship) the NCCAM has not funded any studies of homeopathy, which is true. However, they still accept applications for homeopathic research, but none have made it through the review process and been awarded funding.

This is a tricky issue. Dr. Briggs pointed out that it is not the job of the NCCAM to make final pronouncements about any treatment or medical claim. This is fair enough – but depends on context. The NCCAM is responsible for informing the public about so-called CAM modalities, and that should include a fair assessment of the science. If the science says a treatment is worthless, the NCCAM should not be afraid to say so.

Further, the NCCAM does determine what studies the NCCAM funds. The NCCAM accepts applications for research into homeopathy, but have not funded any in several years. What does this mean? Will they consider funding homeopathy research, and if so they are basically saying that they do not close the door on any medical modality, no matter how implausible or damned by negative evidence.

If they will not consider funding homeopathy, then why are they accepting grant applications for homeopathy research? This could be construed and disingenuous – perhaps a way to not fund homeopathy research without having to say they will not fund homeopathy research.

This leads directly to our final core point of concern.

NCCAM Information

The final major topic of discussion was the information that the NCCAM provides on its website, newsletter, and press releases. In my opinion this is the easiest problem for the NCCAM to address, and one that is completely and solely within their purview – the information they themselves publish.

We were armed with the latest NCCAM newsletter, in which Dr. Briggs is quoted as saying that “Science must be neutral.” Of course, we agree. But in the same newsletter there is article discussing the evidence for acupuncture and pain showing a model of chi and meridians – mystical life force and the lines through which they allegedly flow.

There is also an interview with a member of the NCCAM national advisory board, Xiaoming Tian, a Chinese Medical Doctor. In the article he states that he uses acupuncture to treat a variety of ailments (1. Chronic and acute pain, 2. Osteoarthritis, 3. Fibromyalgia, 4. Sports injuries, 5. Sciatica and neuralgia, 6. Automobile-accident injuries, 7. Autoimmune diseases, 8. Allergies and asthma, 9. Depression, anxiety, and stress, 10. Bell’s palsy and paralysis, 11. Skin rashes and eczema, 12. Side effects of chemotherapy and radiation therapy for cancer.)

The pattern of information is consistent – NCCAM staff talk about a strict adherence to evidence-based medicine and science being neutral, but interspersed with this is an uncritical presentation of ancient superstition as if it were science, and endorsement of treatments that are not backed by science, and in fact have been shown not to work.

It is my interpretation of the evidence that acupuncture has not been shown to work for any indication (as I have written before, the studies show it does not matter where you stick the needles or if you stick the needles, and any benefit appears to be due to placebo effects, artifact, and the non-specific effects of the ritual surrounding acupuncture – none of which constitute acupuncture itself). But I will acknowledge that there can be some reasonable disagreement about whether or not acupuncture is useful for some symptomatic treatment, like pain. The problem is that wishy-washy evidence for symptomatic benefit is then used to support the use of acupuncture for serious medical conditions, like nerve injury. It’s a classic bait and switch.

All of this confirms our worst fears about NCCAM – that its very existence, and the generally positive and uncritical information it provides to the public, is used to promote and endorse unscientific medical modalities.

In fact, it is not enough to be “neutral”, which could easily fall into the trap of false balance (balancing legitimate scientific evidence and analysis with pseudoscientific promotion). The neutrality of science means letting the chips fall where they may – fairly and honestly reporting the state of the evidence without pulling any punches, like the HCSTC did regarding homeopathy.

But it is my experience that the worst thing that the NCCAM will say about a treatment is that there is not “yet” evidence to support its use. The “yet” is often used, but when not it is implied. Almost invariably the lack of evidence leads to the conclusion that “more research is needed.”  What we don’t hear is that there is evidence for lack of efficacy, or a recommendation to not use a modality or to abandon further research.

Given that the CAM community is actively exploiting the existence of the NCCAM as an imprimatur of legitimacy, the NCCAM needs to take special care to avoid such exploitation. Meanwhile, it seems that they go out of their way to encourage such exploitation (although it seems just out of naivete) or at least make it easy.

We pointed out that we do not expect the NCCAM to engage in “debunking” (that’s our job). But we do expect that they are fair and do not give a free pass or special treatment to a modality because it’s CAM. That is the double standard we are frequently complaining about.

On a side note, Dr. Briggs did agree that anti-vaccine sentiments are common in the world of CAM and that the NCCAM can do more to combat this. Information countering anti-vaccine propaganda would be a welcome addition to the NCCAM site.

Conclusion

We greatly appreciate Dr. Briggs giving us the opportunity to voice our concerns to her and her staff directly. The meeting was overall very pleasant and constructive. We hope this will lead to an ongoing dialogue and as a result we can help the NCCAM evolve into a more science-based institution. Dr. Briggs did clearly voice her intention to make NCCAM a more rigorous scientific institution, in line with other centers at the NIH.

The one concrete result of the meeting was an offer to have experts from SBM review NCCAM material before it is published. We, of course, agreed to offer our services.

There continue to be very important issues and questions that are at a “higher level” than the NCCAM itself – such as the optimal regulation of medical products and practices, and also whether or not the public is best served by having a center of funding at the NIH which is organized around such a nebulous concept as CAM, rather than a disease or biological system. SBM will continue to address these issues head on.

But we are also happy to work with the NCCAM, and Dr. Briggs does profess her intention to move the NCCAM in a more rigorous scientific direction. We will see.

Addendum:

In response to my comment that NCCAM fails to condemn ineffective treatments, the following entry on the NCCAM site was pointed out: http://nccam.nih.gov/health/silver/

In which the NCCAM definitely states that colloidal silver does not work and is not safe. The wording of this entry (noting what the FDA states about colloidal silver) reminded my that Dr. Briggs did specifically mention that the NCCAM information is and will be in line with FDA positions on specific products.

To further clarify my statement – the NCCAM and even some CAM promoters in my experience will at times condemn specific products when there is evidence of harm, such as with colloidal silver. But this does not extend to treatment modalities, like homeopathy, acupuncture, or therapeutic touch, nor to mere lack of efficacy.

Further the threshold for negative conclusions about CAM modalities seems to follow a double standard, otherwise chelation for heart disease would never have made it past a review board.

Posted in: Medical Ethics, Politics and Regulation, Science and Medicine

Leave a Comment (58) ↓

58 thoughts on “Our Visit with NCCAM

  1. ScarySkwerl says:

    “Never doubt that a small group of thoughtful, committed people can change the world. Indeed, it is the only thing that ever has.”

    Well done, people.

  2. Jayhox says:

    Nice work. Inch by inch, inroads are being made. Keep it up.

  3. Jojo says:

    Well, that sheds some light on Dr. Atwood’s NCCAM wish list from last week. I’m very happy to hear that Dr. Briggs has initiated this discussion with you guys and that a dialog has been started. That’s certainly a step in the right direction. I’m looking forward to seeing how this plays out.

    As my dad would say, “You done good.”

  4. Scott says:

    It is a bit disappointing that Dr. Briggs declined to comment on anything before her tenure. It’s obviously completely appropriate for her to draw that important distinction, but one would hope that she could still express an intention to definitively break with the past.

    Still, a definite step in the right direction. It must be quite gratifying to get such evidence that you’re having a positive impact on the world.

  5. Michelle B says:

    Cautious enthusiasm is in order. Well done.

  6. roma0104 says:

    While I think most of the stuff is woo, I am willing to let the NCCAM do some testing now and then on alt med if they think it might show something useful. You never know.

    Otherwise, great job guys! It is good to see that someone, somewhere in the alt med community has the guts to sit down and talk about issues in an organized manner, the same cannot be said with Benneth, haha. I hope these relations stay open and good springs out of this.

  7. WilliamLawrenceUtridge says:

    A person starts a new job. On the desk is an envelope from their predecessor. It says “open when you have a problem you can not solve”.

    Time (let’s say….two years) goes by. Such a problem arises, and they open the envelope. Inside is two letters, one labelled “read this first”. It states “Blame everything on the past administration. Do not open the second letter until you have a second problem you are unable to solve.” The person does as the letter directs, and blames everything on the author of the letter.

    More time goes by (let’s say…the average tenure of the head of the NCCAM) and a second problem that can not be solved arises. The person opens the second letter. It says “Write two letters…”

    Passing the buck to the previous administrator isn’t really a good approach. History is part of the context of any system and should be accounted for, but is not a barrier to improving it.

    But it is still very encouraging that she met with the SBM members, and where she appeared to take a stance, it certainly sounded good. I hope we’ll be kept up to date on the actual follow-up.

    Wouldn’t it be sweet if the NCCAM ended up being the primary funder of reserach into the actual effectiveness of CAM, and from there its primary critic?

  8. DevoutCatalyst says:

    “…The one concrete result of the meeting was an offer to have experts from SBM review NCCAM material before it is published. We, of course, agreed to offer our services…”

    Made public, these reviews could be fruitful and will make for interesting reading.

  9. “Wouldn’t it be sweet if the NCCAM ended up being the primary funder of reserach into the actual effectiveness of CAM, and from there its primary critic?”

    I agree. In fact, if the NCCAM lived up to our “wish list” we would probably stop advocating that it be abolished. Specifically if it stopped promoting CAM (or allowing itself to be easily used for this purpose) and funded hard core and ethical efficacy trials, we wouldn’t have a problem. We could then focus on making regulations more science-based.

    This could very much end up like Edzard Ernst’s career – he is a professor of CAM, but strictly science based, and could not help but ultimately come to the conclusion that there is no there there with CAM.

  10. Ian says:

    Gosh I would love it if they became truly a science-based debunking outfit. It would make Harkin’s head spin. :)

  11. art malernee dvm says:

    The one concrete result of the meeting was an offer to have experts from SBM review NCCAM material before it is published. We, of course, agreed to offer our services.>>>>

    Studies show that expert peer review can not be trusted. Studies show except for technical editing the old verify then publish that doctors have used for the last 200 years does not work. Why not let NCCAM go ahead and publish and then verify by making the publication interactive on the internet with not only the “experts” but the “coal faced”? Publish then verify is how its done in other areas of science why not medicine?

  12. “Studies show that expert peer review can not be trusted. Studies show except for technical editing the old verify then publish that doctors have used for the last 200 years does not work.”

    Yes – but were those studies peer-reviewed?

    Seriously – I think you need to clarify this premise. Peer-review is not a panacea, and no guarantee of correctness. But it is an important filter and does improve the quality of the final result.

    In any case – NCCAM literature is not a blog free-for-all. It is meant to be authoritative information, and there is a role for that. Part of what we offer is our nuanced understanding of the deceptive practices of quackery. Dr. Briggs said she does not want NCCAM to be exploited to promote bad science. We can help NCCAM achieve that goal.

  13. I’m also disappointed that Dr. Briggs declined to comment on anything before her tenure. When I take over a new function or position, one of the first things I do is an analysis of the current state of things regarding how they got to where they are and what changes need to be made. For Dr. Briggs to not comment on the state of the NCCAM when she arrived is a cop out.

    We are likely to infer from her comments that either she did not do such an appraisal or was unwilling to disclose the conclusions from her appraisal for one reason or another.

    I can understand that the politically correct thing is usually to not throw your predecessor totally under the buss, but if you take over a organization, it is important to make it clear if you intend to not continue the policies and practices of the previous administration and clarify what will be different under your tenure in charge.

    I would think the better answer would have been, “When I took over the NCAAM, I identified some existing problem areas and practices needing attention, and while I won’t go in to details regarding the things I found when I arrived, TACT is one of those things that pinged my radar.”

  14. @WilliamLawrenceUtridge

    I heard a variation on that story when my previous CIO spoke to the IT dept on his first day.

    The variation went like this:

    The new head of IT finds a letter on his desk on his first day addressed “To the new CIO”. In the letter is the following:

    “1 Blame everything on the old CIO.
    2 Change/replace everything.
    3 Write a letter for the next guy.”

    That CIO lasted 1-1/2 years, 6 months less the his predecessor made it, who lasted 1 year less than the guy who was in charge when I first arrived just after he was hired.

  15. hgkelley says:

    I am so glad the NCCAM invited SBM to give them rational evaluations of the Center’s work. I don’t know whether the photo was taken at the beginning or the end of the meeting, but the body postures of the SBM representatives vs those of the NCCAM staff show who had the most confidence in his/her relative position. I have followed your work for some time now, both on SGU and the blogs, and find your approach to controversial subjects both reasonable and measured, rather than especially confrontational. I also have never heard you make the mistake of attacking the person rather than addressing the subject matter on important issues. Therefore, the defensive or “hiding something” postures of the NCCAM personnel appear, to me, to be a non-verbal expression of a general lack of confidence in their own work to date. I hope that this will spur them to choose a better path in the future.

  16. moderation says:

    It appears to me that the NCCAM has been similar to a lot of other supposed “investigative” bodies (please see: American Academy of Pediatrics Section of Complementary and Integrative Medicine) … in that they claim to be advocating for rigorous CAM research while seriptiscously promoting CAM (weather intentionally or naively). I believe this insinuation into medcine to be a much greater threat to a science based approach than JB Handley, Joseph Mercola, Dana Ullman or any other anti-vaxxer / snake oil salesman.

  17. Mojo says:

    “In fact, if the NCCAM lived up to our “wish list” we would probably stop advocating that it be abolished.”

    But perhaps others would start. You mention Edzard Ernst; as a result of his science based approach, various apologists for CAM (including some in high places) have been campaigning against him for some time.

  18. TimonT says:

    Isn’t part of the problem that HIH programs must curry the favor of their Congressional Overlords (i.e. where the money comes from)? Do you think this is a factor in what Dr. Briggs and her colleagues feel they can say/do? Would it be appropriate for them to consider one of their tasks to be to educate members of Congress?

    For example, I saw a video of a hearing of Sen. Harkin’s Health committee with Dr. Insel of the NIMH. It was clear that Dr. Insel was carefully modulating his language, especially in response to Harkin’s question about autism and vaccines. Is it conceivable that someone from NIH could meet personally with Harkin and try to educate him about medical science (probably impossible, I realize)?

    Congratulations on your most excellent efforts!

  19. WilliamLawrenceUtridge says:

    Part of the issue that Dr. Briggs faced as well, is that with most scientific organizations I would think you can more or less trust your constituents – though people will pimp their own research, you can trust other researchers during peer review for funding (which I’m guessing is much more rigorous than the review for publication) to point out holes, failings and general issues.

    Can’t do that with CAM. The culture is about insulating itself from criticism, avoiding actual scrutiny, denying findings rather than following-up on them, confirming rather than challenging bias, relying on anecdote instead of data, statistical fishing and mining for positive results instead of correcting for them, divergent hypotheses rather than convergent evidence. You’ve got a bunch of reviewers whose central question is “will a positive result for this trial make us look good?” rather than “will this really test the intervention to inform real-life practice, decision making or basic medical knowledge?”

    Chances are in any of the other institutes, you could make controversial decisions but at least rely on your predecessor to have made an honest go of it based on the information they had and a common scientific approach and mindset, using peer-review as an honest way of challenging studies to ensure the money is well-spent. You don’t have to second-guess the fact that you’re funding science, even if you do have to second-guess the fact that you’re funding the best science. You might be criticized for making the wrong choice, but at least your approach wouldn’t be embarrassing, wrong-headed and ultimately little more than wishful thinking.

  20. anatotitan says:

    Heroic. Thank you.

  21. Studies show that expert peer review can not be trusted. Studies show except for technical editing the old verify then publish that doctors have used for the last 200 years does not work. Why not let NCCAM go ahead and publish…

    In addition to Steve’s answer, please consider that whatever the result of our pre-publication reviews may be, they will be newsworthy: if our recommendations are followed, great. If not, that tells us something, too, and we will be certain to write about it.

  22. qetzal says:

    art malernee dvm:

    Studies show that expert peer review can not be trusted.

    It’s certainly not perfect, and should never be blindly trusted. That doesn’t mean it’s worthless, as you seem to be implying.

    Why not let NCCAM go ahead and publish and then verify by making the publication interactive on the internet with not only the “experts” but the “coal faced”?

    We’ve already got ‘publish without verification’ in spades, with the likes Mercola, whale.to, etc. We don’t need more of the same from NCCAM.

    Publish then verify is how its done in other areas of science why not medicine?

    Because that’s not how it’s done in other areas of science. (Esp. not life sciences.)

  23. David Gorski says:

    Isn’t part of the problem that HIH programs must curry the favor of their Congressional Overlords (i.e. where the money comes from)? Do you think this is a factor in what Dr. Briggs and her colleagues feel they can say/do? Would it be appropriate for them to consider one of their tasks to be to educate members of Congress?

    For example, I saw a video of a hearing of Sen. Harkin’s Health committee with Dr. Insel of the NIMH. It was clear that Dr. Insel was carefully modulating his language, especially in response to Harkin’s question about autism and vaccines. Is it conceivable that someone from NIH could meet personally with Harkin and try to educate him about medical science (probably impossible, I realize)?

    Absolutely. One thing you have to remember (and that Dr. Briggs mentioned at our meeting) is that, for all his support of quackery, Senator Harkin is viewed as a champion of the NIH, and, if you leave his role in creating NCCAM out of consideration, probably rightly so. Consequently, it’s not just NCCAM that curries favor with him.

    In Harkin’s defense (a rare thing from me), I don’t see any evidence that he is anti-vaccine, like, for example, Representative Dan Burton. Sen Harkin may be a bit too credulous listening to the alt-med types who are anti-vaccine, but as far as I can tell he doesn’t buy into the anti-vax misinformation.

  24. David Gorski says:

    Can’t do that with CAM. The culture is about insulating itself from criticism, avoiding actual scrutiny, denying findings rather than following-up on them, confirming rather than challenging bias, relying on anecdote instead of data, statistical fishing and mining for positive results instead of correcting for them, divergent hypotheses rather than convergent evidence. You’ve got a bunch of reviewers whose central question is “will a positive result for this trial make us look good?” rather than “will this really test the intervention to inform real-life practice, decision making or basic medical knowledge

    Perhaps so, but Dr. Briggs has taken a step in the right direction with regard to peer review by moving most peer review of NCCAM grants to standing study sections in the Center for Scientific Review, just like most other NIH grants. The members of such study sections are not chosen by NCCAM; they are chosen by CSR, which is currently run by my old chairman from graduate school, Tony Scarpa.

  25. Joe says:

    It is great to read that the NCCAM is going to pay attention to criticism. However, I think Dr. Briggs was wrong to dodge the TACT study since they are still providing funding. Along the lines of Karl W’s comment, I think she could have looked at pointless expenses and cut them. She has had two years to do that, and the the problems with it must certainly have come to her attention a while ago.

  26. edgar says:

    I saw her speak at APHA in November, and was pretty impressed with what she had to say.

  27. Saffron says:

    Bravo.

  28. chaos4zap says:

    Most days, I am concerned about the future of medicine and science. Each time I read this blog, I get a little less concerned. Knowing that there are dedicated professionals out there giving their free time to fight the good fight gives me (and I’m sure many others) hope that the collective public will one day appreciate the methodology in science that has given us so many things we take for granted today. It puts a smile on my face to see that your efforts are being noticed by such organizations as the NCCAM. What I mean by that is, knowing what I do about the workings of the NCCAM in the past, I never in a million years would have wagered that anyone over there was an avid reader of SBM, much less, willing to extend an invite for constructive conversation. I was beginning to wonder if they even had a concept of what science based medicine was or its importance. I cannot applaud enough for the dedication and time all contributors to this blog put in. You may already know this by now, but there are many of us lay-people that truly look up to all of you and your efforts. You are quickly becoming the “tip of the spear” for reason and logic. The world is a better place because of your passion. Thank you.

  29. TimonT says:

    Dr. Gorsky, thanks for your response to my comment.

    And I obviously meant “NIH, not “HIH”. (Why can’t these damn spell checkers be smarter?)

  30. Angora Rabbit says:

    Congratulations on being recognized by NCCAM as a “constituent” worthy of consideration (to use the NIH’s own buzzwords). Glad to learn that your views are being heard and you now have this formal opportunity to shape NCCAM policy. Good luck with it and thanks.

    NIH has initiated an institute-by-institute review of mission, and there is a movement to consolidate institutes “when appropriate” to reduce costs and enhance collaboration. We can have a long discussion on whether this is a good or bad idea. But I think that your persistent drumbeat in this environment is getting NCCAM’s attention. If there are constituents protesting their mission and ability, this can factor into when and if their review occurs. Not to belittle what you’ve done, but rather to say “bravo” and “good timing.” The transfer of TACT trial from NCCAM to NHLBI could be seen as a black mark against NCCAM if it is shifting the trial to another institute’s portfolio.

    Regarding whether NCCAM would refuse proposals on a topic, readers may find it helpful to understand that the Institutes and Program Officers are rewarded for the number of proposals that are submitted to a particular PO’s portfolio. There is no incentive to refuse them. As long as this policy continues, it is unlikely that NCCAM will turn down applications, simply because it makes them look important and useful. The culture at NIH extramural will have to change before that happens.

    Horrifying to know that only recently (if I understand this correctly) that review of NCCAM proposals has only recently been shifted to the CSR study sections. As a longtime CSR reviewer, I’m now going to blow on the dice and hope to get lucky. :)

  31. It’s nice to learn that this site isn’t just a prophet crying out in the wilderness unheard, but is actually attracting serious attention.

    This site is clearly more than either an echo chamber or a troll feeding station. SBM is getting the message out and having an impact.

    Bravo SBM!

  32. art malernee dvm says:

    Yes – but were those studies peer-reviewed?>>>> your joke makes my point about prepublication peer review by experts.

    Prepublication peer review is faith based not evidence based

    Richard Smith: Scrap peer review and beware of “top journals”
    22 Mar, 10 | by julietwalker

    The neurologist and epidemiologist Cathie Sudlow has written a highly readable and important piece in the BMJ exposing Science magazine’s poor reporting of a paper on chronic fatigue syndrome, (1) but she reaches the wrong conclusions on how scientific publishing should change.

    For those of you who have missed the story, Science published a case control study in September that showed a strong link between chronic fatigue syndrome and xenotropic murine leukaemia virus-related virus (XMRV). (2) The study got wide publicity and was very encouraging to the many people who believe passionately that chronic fatigue syndrome has an infectious cause. Unfortunately, as Sudlow describes, the study lacked basic information on the selection of cases and controls, and, worse, Science has failed to publish E-letters from Sudlow and others asking for more information.

    In the meantime, three other studies have not found an association between chronic fatigue syndrome and XMRV. (3-5)

    To avoid such poor reporting in the future Sudlow urges strengthening the status quo—more and better prepublication peer review. Not only is she trying to close the stable door after the horse has bolted she has also failed to recognise the possibilities of the new Web 2.0 world. The time has come to move from a world of “filter then publish” to one of “publish then filter”—and it’s happening.

    Prepublication peer review is faith based not evidence based, and Sudlow’s story shows how it failed badly at Science. Her anecdote joins a mountain of evidence of the failures of peer review: it is slow, expensive, largely a lottery, poor at detecting errors and fraud, anti-innovatory, biased, and prone to abuse. (6 7) As two Cochrane reviews have shown, the upside is hard to demonstrate. (8 9) Yet people like Sudlow who are devotees of evidence persist in belief in peer review. Why?

    The world also seems unaware that it is scientifically dangerous to read only the “top journals”. As Neal Young and others have argued, the “top journals” publish the sexy stuff. (10) The unglamorous is published elsewhere or not at all, and yet the evidence comprises both the glamorous and the unglamorous.

    The naïve concept that the “top journals” publish the important stuff and the lesser journals the unimportant is simply false. People who do systematic reviews know this well. Anybody reading only the “top journals” receives a distorted view of the world—as this Science story illustrates. Unfortunately many people, including most journalists, do pay most attention to the “top journals.”

    So rather than bolster traditional peer review at “top journals,” we should abandon prepublication review and paying excessive attention to “top journals.” Instead, let people publish and let the world decide. This is ultimately what happens anyway in that what is published is digested with some of it absorbed into “what we know” and much of it never being cited and simply disappearing.

    Such a process would have worked better with the story that Sudlow tells. The initial study would have appeared–perhaps to a fanfare of publicity (as happened) or perhaps not. Critics would have immediately asked the questions that Sudlow asks. Instead of hiding behind Science’s skirts as has happened, the authors would have been obliged to provide answers. If they couldn’t, then the wise would disregard their work. Then follow up studies could be published rapidly.

    Unfortunately, unlike physicists, astronomers, and mathematicians, all of whom have long published in this way, biomedical researchers seem reluctant to publish without traditional prepublication peer review. In reality this is probably because of innate conservatism and the grip of the “top journals” who insist on prepublication review, but biomedical researchers often say “But our stuff is different from that of physicists in that it may scare ordinary people. A false story, for example, “Porridge causes cancer” can create havoc.”

    My answer to this objection is that this happens now. Much of what is published in journals is scientifically poor—as the Science article shows. Then, many studies are presented at scientific meetings without peer review, and scientists and their employers are increasingly likely to report their results through the mass media.

    In a world of “publish then filter” we would at least have the full paper to dissect, whereas reports in the media even if derived from scientific meetings, include insufficient information for critical appraisal.

    So I urge Sudlow, a thinking woman, to reflect further and begin to argue for something radical and new rather than more of the same.

    1. Sudlow C. Science, chronic fatigue syndrome, and me. BMJ 2010;340:c1260

    2. Lombardi VC, Ruscetti FW, Das Gupta J, Pfost MA, Hagen KS, Peterson DL, et al. Detection of an infectious retrovirus, XMRV, in blood cells of patients with chronic fatigue syndrome. Science 2009;326:585-9.

    3. Van Kuppeveld FJM, de Jong AS, Lanke KH, Verhaegh GW, Melchers WJG, Swanink CMA, et al. Prevalence of xenotropic murine leukaemia virus-related virus in patients with chronic fatigue syndrome in the Netherlands: retrospective analysis of samples from an established cohort. BMJ 2010;340:c1018.

    4. Erlwein O, Kaye S, McClure MO, Weber J, Willis G, Collier D, et al. Failure to detect the novel retrovirus XMRV in chronic fatigue syndrome. PLoS One 2010;5:e8519.

    5. Groom HC, Boucherit VC, Makinson K, Randal E, Baptista S, Hagan S, et al. Absence of xenotropic murine leukaemia virus-related virus in UK patients with chronic fatigue syndrome. Retrovirology 2010;7:10.

    6. Godlee F, Jefferson T. Peer Review in Health Sciences. 2nd ed. London: BMJ Books; 2003.

    7. Smith R. Peer review: A flawed process at the heart of science and journals. J R Soc Med 2006;99:178-182.

    8. Jefferson T, Rudin M, Brodney Folse S, Davidoff F. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database of Systematic Reviews 2007, Issue 1. Art. No.: MR000016. DOI: 10.1002/14651858.MR000016.pub3

    9. Demicheli V, Di Pietrantonj C. Peer review for improving the quality of grant applications. Cochrane Database of Systematic Reviews 2007, Issue 1. Art. No.: MR000003. DOI: 10.1002/14651858.MR000003.pub2

    10. Young NS, Ioannidis JPA, Al-Ubaydli O, 2008 Why Current Publication Practices May Distort Science. PLoS Med 5(10): e201. doi:10.1371/journal.pmed.0050201

    Competing interest: RS is on the board of the Public Library of Science and an enthusiast for open access publishing, but he isn’t paid and doesn’t benefit financially from open access publishing.

  33. Harriet Hall says:

    For those who may have missed it, I wrote about the retrovirus/CFS study at http://www.sciencebasedmedicine.org/?p=3932

  34. art malernee dvm on abolishing peer review:

    “let people publish and let the world decide.”

    Nobody’s stopping them. They can get a WordPress blog for free like anybody else and go to town.

    “Such a process would have worked better with the story that Sudlow tells. The initial study would have appeared—”

    Where? There’s no reason to think that it would have appeared in Science. If Science published anything people sent them without reading it first there might not have been room for the Chronic Fatigue article. But without peer review, journals lose their prestige anyway. So presumably they would have skipped Science completely and gone directly to WordPress.

    Ok. So they publish the paper on their blog and call all the journalists they know to announce that they have a new blog post. (Just like they could do now anyway.)

    “… perhaps to a fanfare of publicity (as happened) or perhaps not. Critics would have immediately asked the questions that Sudlow asks. Instead of hiding behind Science’s skirts as has happened, the authors would have been obliged to provide answers. If they couldn’t, then the wise would disregard their work.”

    Which is different from what is actually happening how, exactly? Critics are asking questions, the wise are disregarding their work. Your point is… what?

  35. art malernee dvm says:

    Your point is… what?>>> I think the reason prepublication peer review is faith based not evidence based is because we all overestimate our own knowledge and skills. Its the way our minds work.

  36. art malernee dvm on self-assessment:

    “we all overestimate our own knowledge and skills.”

    No, only incompetent and ignorant people do. They don’t know enough to know that they don’t know. They don’t know what competence looks like. So they overestimate themselves relative to others.

    Competent and educated people are fully aware of the limitations of their own knowledge and abilities. They have pushed them to the edge and know where they end. On the other hand, things they are competent at feel easy to them, so they think they are easy and that anyone could do them. Therefore they are likely to underestimate their knowledge and abilities relative to others.

    Competence and education actually do exist, objectively. Professional athletes really are better at sports than I am, and they really do know more about physical training and the world of competitive sport.

    Professional scientists know more about the practice of science than laypeople do, and they do better than chance at distinguishing well-conducted research from blathering. They aren’t perfect, but if (say) David Gorski tells me that such-and-such is a good research paper on breast cancer surgery I am going to give him more credence than whatever I get by entering “breast cancer surgery” in Google and rolling dice to decide which hit to look at.

  37. weing says:

    While not perfect, I find peer-review very useful. It serves as an initial filter. Who has the time to read all the studies published? This way, there is less studies to read and dismiss yourself. Sometimes they get sloppy and it gets noticed. Sometimes, they may be doing similar research and could be tempted to delay publication of a study that would scoop them. What can I say, we live in an imperfect world.

  38. qetzal says:

    art malernee dvm:

    I think the reason prepublication peer review is faith based not evidence based is because we all overestimate our own knowledge and skills. Its the way our minds work.

    But of course, you’re not overestimating your own knowledge of peer review, right?

  39. JMB says:

    Congratulations to the SBM faculty. Maybe there’s hope that some wasted taxpayer money can be saved by using a science based approach.

    In regards to the peer review issue, both approaches can coexist (filter by peer review versus publish and review afterward). Traditional print media is better suited to the prepublication review. Publishing in electronic media is better suited to publish and review. However, in the electronic media approach there would need to have some way to publish criticism or support in a blog associated with the article. There would still need to be standards imposed on authors in the publication and associated blog to disclose financial interest and establish credentials of authors and respondents. Any published experiments would have to clearly cite the IRB approval. Finally, any public website would have to provide a basic discussion cautioning the public about how to interpret the scientific (or not so scientific) data.

    Criticism of the peer review process is founded, but in spite of its limitations, it does force submitted articles to conform to some minimum standards of scientific study. Furthermore, there is no way the print media can publish all of the papers submitted for review.

    While the faculty of SBM may not have the resources to research whether studies were conducted with the oversight of an Institutional Review Board, verify calculation of statistics, or verify financial disclosures, I doubt that will often be needed for peer review of the NCCAM newsletter or website. I hope SBM will be asked to peer review the NCCAM newsletter and website. I do think that a government agency should provide resources to investigate financial disclosures.

    I think editors of peer reviewed journals should take more responsibility in recognizing when certain articles may produce a significant public response. The publication of the article on XMRV in Science will result in millions of dollars spent before we have a solid scientific basis. Editors do pick who will do the peer review for a submitted article, and they know which reviewers will be easy, or more stringent. They certainly bear some responsibility for the publication of an article. When an article is likely to have a significant impact, there should be a higher standard of peer review. Resources should be provided to confirm financial disclosures, a statistician should be employed to critique the statistical design and calculations, and peers should be selected based on expertise, but from different fields of science (such as virologists and infectious disease specialists). Perhaps the names of the peer reviewers should be published in such an public impact study.

  40. Versus says:

    @ “This is a tricky issue. Dr. Briggs pointed out that it is not the job of the NCCAM to make final pronouncements about any treatment or medical claim.”

    This seems to be a bit of a verbal dodge. No one is asking for “final pronoucements.” Just give us the evidence — if there is none, and it doesn’t seem likely there ever will be, simply say so and use NCCAM’s resources for other purposes.

    And where does Dr. Briggs get this idea in the first place? The federal law creating NCCAM certainly doesn’t say this. The purpose of NCCAM is, by statute, “the conduct and support of basic and applied research (including both intramural and extramural research), research training, the dissemination of health information, and other programs with respect to identifying, investigating, and validating complementary and alternative treatment, diagnostic and prevention modalities, disciplines and systems.” 42 U.S.Code Sec. Sec. 287c-21.

    Despite what Sen. Harkin may say, I don’t think a fair reading of the statute means that NCCAM must, by law, actually validate CAM no matter what the research says. In addition, the statute specifically states that “the Director of the Center shall identify and evaluate alternative and complementary medical treatment, diagnostic and prevention modalities in each of the disciplines and systems with which the Center is concerned . . .” Id.

    “Evaluate” means “to determine the significance or quality of; assess.” (Random House Webster’s College Dictionary). Unless”evaluate” has a different meaning in medicine, it appears the statute gives full authority to NCCAM to determine the quality of the evidence in suppport of a particular procedure and, if there is none, to say so, and to refuse to spend any more resources on it.

    In sum, it appears NCCAM could do everything you asked it to and still be in complicance with the law creating it.

  41. JMB says:

    Having done peer review, I can say it is not faith based. The peer reviewer writes a letter (similar to the blogs here) that is reviewed by the editor/s. There is usually more than one peer reviewer. When a peer reviewer is dismissing the article without a sound science based methodologic argument, the article tends to get sent to another reviewer.

  42. daijiyobu says:

    I’m pretty sure the irony isn’t intentional, but MD Benda who is big in the AANP circle has a post up at AANP’s usually vapid blog “Physicians Who Listen” titled

    “Horsing Around at the NIH”

    (see http://naturopathicphysicians.blogspot.com/2010/04/horsing-around-at-nih.html )

    I particularly like

    “[Benda has] published two prior research studies (one part of an NCCAM Center grant) investigating the effect of hippotherapy (physical therapy on horseback) on children with spastic cerebral palsy.”

    -r.c.

  43. DREads says:

    Studies show that expert peer review can not be trusted

    What study? How could a study invalidate peer review? That does not make any sense. Certainly, a study could look into sociological and institutional factors that bias or contaminate the peer review process. No reasonable person would argue that peer review is perfect or infallible. But eliminate it? Without peer review, there’s no accountability. Peer review helps ensure ideas withstand scrutiny. Scientists are not perfect and feedback can help them catch for errors and improve their research. When evidence is lacking, it is harder to build consensus, and more research is needed. Even in the face of politics, good ideas should eventually prevail given a critical mass of evidence.

    The level of effort needed to discover new ideas that withstand intellectual scrutiny can be daunting. Not everyone can handle the stress on a constant basis. It is perfectly reasonable to take a vacation or to leave science permanently. However, avoiding peer review is not an acceptable alternative.

  44. The Dicklomat says:

    If The US Patent Office can reject-with-prejudice all submissions for “free energy” devices without even looking at the specs (on the basis of the known laws of thermodynamics), then NCCAM can outrightly reject applications for Homeopathy for similar reasons and with similar disdain.

  45. Wait, I figured it out. art malernee dvm thinks we are choosing between two possible sequences:
    - verify > publish or
    - publish > verify.

    Since three peer reviewers can never be perfect, he prefers publish-then-verify.

    Except that the final verify will always be there. No matter what the practices of editors, scientists always critique and verify. That never goes away. So in fact we are choosing between the sequences
    - filter > publish > verify or
    - publish > verify.

    Opponents of peer review are proposing that we do away with the filter, not the verification. I happen to think a filter is useful.

    PubMed carries about 5,000 journals; if none of the 5,000 accept a particular paper for publication, that probably says something about the paper. Once the paper has been published in one of the listed journals, anyone who needs the information can search for it. If I am looking for obscure technical information I’m not going to find it in Science, but that’s fine: I am going to find it.

  46. caoimh says:

    Great job SBM!

    Keep up the good work.

  47. art malernee dvm says:

    Studies show that expert peer review can not be trusted
    What study? How could a study invalidate peer review? That does not make any sense>>>>

    Editorial peer review for improving the quality of reports of biomedical studies

    Jefferson T, Rudin M, Brodney Folse S, Davidoff F

    Summary
    Editorial peer review for improving the quality of reports of biomedical studies
    Editorial peer review is used world-wide as a tool to assess and improve the quality of submissions to paper and electronic biomedical journals. As the information revolution gathers pace, an empirically proven method of quality assurance is of paramount importance. The increasing availability of empirical research on the possible effects of peer review led us to carry out a review of current evidence on the efficacy of editorial peer review. We found few studies of reasonable quality, and most of these were concerned with the effects of blinding reviewers and/or authors to each others’ identity. We could not identify any methodologically convincing studies assessing the core effects of peer review. Major research is urgently needed.

    This is a Cochrane review abstract and plain language summary, prepared and maintained by The Cochrane Collaboration, currently published in The Cochrane Database of Systematic Reviews 2010 Issue 3, Copyright © 2010 The Cochrane Collaboration. Published by John Wiley and Sons, Ltd.. The full text of the review is available in The Cochrane Library (ISSN 1464-780X).
    This record should be cited as: Jefferson T, Rudin M, Brodney Folse S, Davidoff F. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database of Systematic Reviews 2007, Issue 2. Art. No.: MR000016. DOI: 10.1002/14651858.MR000016.pub3.

    This version first published online: October 20. 2003
    Last assessed as up-to-date: February 20. 2007

    Abstract
    Background
    Scientific findings must withstand critical review if they are to be accepted as valid, and editorial peer review (critique, effort to disprove) is an essential element of the scientific process. We review the evidence of the editorial peer-review process of original research studies submitted for paper or electronic publication in biomedical journals.

    Objectives
    To estimate the effect of processes in editorial peer review.

    Search strategy
    The following databases were searched to June 2004: CINAHL, Ovid, Cochrane Methodology Register, Dissertation abstracts, EMBASE, Evidence Based Medicine Reviews: ACP Journal Club, MEDLINE, PsycINFO, PubMed.

    Selection criteria
    We included prospective or retrospective comparative studies with two or more comparison groups, generated by random or other appropriate methods, and reporting original research, regardless of publication status. We hoped to find studies identifying good submissions on the basis of: importance of the topic dealt with, relevance of the topic to the journal, usefulness of the topic, soundness of methods, soundness of ethics, completeness and accuracy of reporting.

    Data collection and analysis
    Because of the diversity of study questions, viewpoints, methods, and outcomes, we carried out a descriptive review of included studies grouping them by broad study question.

    Main results
    We included 28 studies. We found no clear-cut evidence of effect of the well-researched practice of reviewer and/or author concealment on the outcome of the quality assessment process (9 studies). Checklists and other standardisation media have some evidence to support their use (2 studies). There is no evidence that referees’ training has any effect on the quality of the outcome (1 study). Different methods of communicating with reviewers and means of dissemination do not appear to have an effect on quality (3 studies). On the basis of one study, little can be said about the ability of the peer-review process to detect bias against unconventional drugs. Validity of peer review was tested by only one small study in a specialist area. Editorial peer review appears to make papers more readable and improve the general quality of reporting (2 studies), but the evidence for this has very limited generalisability.

    Authors’ conclusions
    At present, little empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research. However, the methodological problems in studying peer review are many and complex. At present, the absence of evidence on efficacy and effectiveness cannot be interpreted as evidence of their absence. A large, well-funded programme of research on the effects of editorial peer review should be urgently launched.

  48. Ash says:

    I don’t think that study (based on the abstract) says that peer review is a waste of time. In particular it seems to focus on editorial peer review, as opposed to peer review of the methodology and interpretation of the results.

  49. JMB says:

    Part of quotes provided by Art Malernee DVM:
    “We found few studies of reasonable quality, and most of these were concerned with the effects of blinding reviewers and/or authors to each others’ identity. We could not identify any methodologically convincing studies assessing the core effects of peer review.”

    Sentence 1 refers to the practice of withholding the name of the author from the peer reviewer to avoid popularity or reputation being a reason to accept or reject an article, and to avoid the author lobbying or retaliating against the reviewer. Why that would be subject to scientific study, I don’t know.

    Sentence 2 states there are no studies published that can convincingly show a positive or negative effect on the quality of published studies (the core effects) due to the peer review process.

    “Authors’ conclusions
    At present, little empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research. However, the methodological problems in studying peer review are many and complex. At present, the absence of evidence on efficacy and effectiveness cannot be interpreted as evidence of their absence.”

    The last sentence is pretty straightforward. The sentence #2 clearly explains why that is the case. To top it off, they are using peer review of the peer review process to determine results. Positive or negative effects may be observed if you change which peer review group is assigned as the gold standard.

    Citation #7
    “7. Smith R. Peer review: A flawed process at the heart of science and journals. J R Soc Med 2006;99:178-182.”
    This is an opinion piece submitted by a CEO of a health insurance company. For most doctors, that is reason enough to question credibility (sorry for my bias, but insurance companies’ denials certainly indicates a ruthless profit motive).

    While you can say that there is no evidence that the peer review process has an overall positive effect on medical science, you cannot say that consequently it has a negative effect.

    One simple point is that not all that we call knowledge is based on information that is suitable for scientific empirical investigation.

    It would not surprise me at all if the Duesberg’s HIV/AIDS paper set to be published in Medical Hypothesis was not first submitted to one or more peer reviewed journals. Here is the link to the discussion of that article.

    http://www.sciencebasedmedicine.org/?s=speculative+medicine

  50. StatlerWaldorf says:

    This is great work, and I commend you all for being respectful and very productive. It is meaningful to see real and tangible efforts for change with people working together rather than completely opposed to each other.

  51. art malernee dvm says:

    While you can say that there is no evidence that the peer review process has an overall positive effect on medical science, you cannot say that consequently it has a negative effect.>>>>

    Shouldn’t that read

    you cannot say that consequently it has a overall negative effect.

    not

    ” you cannot say that consequently it has a negative effect.”

  52. JMB says:

    Art Malernee DVM wrote,

    “Shouldn’t that read

    you cannot say that consequently it has a overall negative effect.”

    Agreed. Mistakes have definitely been made in peer review.

  53. Mandos says:

    By the way, this post has been Noticed by the The Integrator blogger (a CAM lobbyist of some sort), who wonders why Briggs would meet with someone who wants to kill her department and its mission. So you are making them worry.

  54. mdcatdad says:

    It seems that NCCAM is conflicted; if it declares something ineffective it cannot accept any more studies for the treatment/modality and thus its budget justification would have to be correspondingly reduced.

    Eventually the Center would zero itself out if it followed the evidence.

  55. Scott Young says:

    You might be interested in viewing Dr. Briggs’ talk yesterday about NCCAM. I’m glad it is only a sliver of the NIH budget.

    http://videocast.nih.gov/Summary.asp?File=15820

  56. Versus says:

    Dr. Briggs’s take on your meeting:

    Director’s Page
    Josephine P. Briggs, M.D.
    Message from the Director:
    Listening to Differing Voices
    April 19, 2010

    http://nccam.nih.gov/about/offices/od/director.htm

    Notice the references to “products” and “natural products” in the discussion of priorities — no mention of “procedures.” Also, wonder what the “compelling” reasons for studying CAM are?

Comments are closed.