The Need for Transparency

A recent editorial in PLOS Medicine discusses the need for transparency in the medical literature, specifically with regard to comparative effectiveness research (CER). The editorial makes many excellent points, but also puts into clear relief the double standard that is consciously being fabricated by proponents of non-science-based medicine.

I wrote previously about another editorial that took a very different approach. Speaking for The Kings Fund, Professor Dame Carol Black said.

“The challenge is to develop methods of research that allow us to assess the value of an approach that seeks to integrate the physical intervention, the personal context in which it is given, and non-specific effects that together comprise a particular therapy.”

The editorial essentially defended the use of CER and other forms of evidence to bolster the evidence base for so-called CAM (complementary and alternative medicine) in order to promote its use.

What I find most interesting is that when medical scientists write in the literature about medical research and practice, absent the context of CAM, their statements and perspective are unapologetically rational and scientific. They typically will advocate for a high standard of science and evidence in making medical policy and decisions. In the context of CAM, however, this standard vanishes and we encounter squirrely apologetics for lowering the scientific standard.

Read the PLOS Medicine editorial and apply the principles they endorse to CAM, and I think you will see what I mean.

With that in mind, let’s turn to the content of the PLOS Medicine editorial, by the journal’s editors. They emphasize the need for publishing negative studies, and lament the “file drawer effect” that is known to bias the published literature toward positive studies. they write:

…the demonstration that an existing practice is ineffective or potentially harmful can (or should) prompt a rapid change in research agendas, policy, and clinical care. Knowing what doesn’t work is particularly useful in efforts to control medical spending, where redirecting limited resources away from ineffective interventions is of obvious benefit.

This is a cornerstone of science-based medicine – actually responding to quality published evidence by altering practice, including stopping practices that are shown not to work or to have unacceptable risks. While mainstream medicine can be criticized for ignoring this rule in some specific cases (while generally applying it), the culture of CAM seems to be based upon ignoring it.

The authors then get to the primary point of their article -that comparative effectiveness research (CER), which has recently received a great deal of funding, requires just as much transparency as traditional efficacy trials in order to ward off distortion of the medical literature.

In short, CER is designed to give real-world comparisons of the applications of various medical interventions (whether diagnostic, preventive, therapeutic, etc.). Such trials are not designed to determine efficacy. I wrote previously about the fact that CER type studies are being abused by some to draw efficacy conclusions – but this is not what such trials are designed to do. For example, they are rarely blinded and do not control for many variables.

But they are useful in assessing the effect on the health care system of various interventions. They just need to be used appropriately.

The PLOS Medicine editors warn that CER has a higher potential for distortion than efficacy trials. They write:

In CER, which will include studies of many types, practices that distort the scientific evidence base—such as ‘‘cherry picking’’ for publication only those studies describing a desired outcome, or ‘‘fishing’’ from an ocean of possible analyses only those that might support favorable (but statistically invalid) conclusions—have the potential to affect policy, practice, and profits to an even greater degree than they have done in the context of traditional efficacy trials.

I agree with their assessment that this means the need for transparency in CER is great. But is also highlights one of the reasons why CER should not be used to determine efficacy, especially of controversial or implausible health claims – and especially as a method to rescue modalities which have failed in efficacy trials.

That is, however, exactly the new approach Dame Carol Black and others who promote CAM have been suggesting.


CER is a useful area of medical research that will help us to navigate toward more cost effective medicine. However, because of the nature of such research, as the PLOS Medicine editors point out, the potential for distortion is great and therefore we need both high standards in research and transparency to minimize known sources of distortion.

For example they specifically recommend (and I agree) extending to CER the requirement to register all clinical trials prior to their onset, so that all results will be made public, not just those favorable to those funding or carrying out the study or those profitable to the study publisher.

They also take the opportunity to endorse open-access journals, which can be seen as a bit self-serving since PLOS is a major open-access medical journal. But their point is legitimate, as open-access journals remove the profit motive from deciding which studies to publish, which creates a motivation to publish positive studies that will garner readers, media attention, and requests for reprints.

In addition, although perhaps not expressly intended by the authors, their points also can be used to argue that CER should not be used to make efficacy claims in the absence of quality efficacy research showing that a specific intervention works.

And finally, the high standards of science and evidence that is overtly being recommended by the PLOS Medicine editors should apply to all of medicine – a single, fair, and thoughtful science-based standard. The call for a double standard or the weakening of the scientific standard of medicine, in order to promote preferred modalities that have not been able to demonstrate efficacy, should be vigorously opposed.

Posted in: Science and Medicine

Leave a Comment (6) ↓

6 thoughts on “The Need for Transparency

  1. mattyv says:

    Thanks for the informative and thoroughly readable (to the non-Doctor) post Steve. It seems everywhere I look right now, somebody is trying to push an agenda using compromised evidence standards. Thus, your points are highly applicable to areas other than medicine. So thanks for helping express some very important ideals that others need to take note of :-)

    ps. The links in the post need some fixing.

  2. Squillo says:

    Concur with mattyv. Thanks for spelling out the incredible double-standard that seems to becoming acceptable with regard to CAM.

  3. pmoran says:

    “The call for a double standard or the weakening of the scientific standard of medicine, in order to promote preferred modalities that have not been able to demonstrate efficacy, should be vigorously opposed.”

    As you point out, there is only a double standard when the parties involved in CER research forget what the question was.

    The question being asked here, after some decoding of what was actually said, is “Even if (this CAM approach) is mainly placebo, can it produce worthwhile, safe, cost-effective clinical outcomes within the present framework of medical practice?”

    This is actually a legitimate question. We suspect that the answer is mostly no, but we cannot on the present evidence assume that the answer is no. Medicine is complex, and the research unclear.

    And, again, look at —

    Linde K, Allais G, Brinkhaus B, Manheimer E, Vickers A, White AR. Acupuncture for migraine prophylaxis. Cochrane Database of Systematic Reviews 2008, Issue 2. Art. No.: CD001218. DOI: 10.1002/14651858.CD001218.pub2

  4. Recall the nature of placebo, which Dr. Novella has written about extensively: “the” placebo effect is just the appearance of efficacy where there is none, based substantially on confusion about the data, on a mess of research artifacts and data noise.

    It is not a pseudo-therapeutic phenomenon derived from the raw power of mind over matter. It is not an untapped medical resource. It is not fairy dust that we can sprinkle on public health policy. Anything that is admitted to be “mainly placebo” is “mainly an illusion.”

    So if we rephrase that question again, we get: “Even if (this CAM approach) is largely based on the illusion of efficacy, can it produce anything worthwhile?”

  5. pmoran says:

    Paul, I am by no means the only battle-hardened skeptic who cannot squeeze all the evidence regarding placebo responses into the “it’s all illusion” box.

    It bothers me that such a final judgment should be made on an evidence base that is inconclusive, if not actually leaning the other way. Is bias leading otherwise good scientists to leave normal scientific caution behind in their eagerness to cut the ground completely from under despicable CAM, folk and hobbyist medical practices?

    There are areas of medicine where placebo influences do not apply in the slightest. But we are also at a point in the history of medicine where there is considerable unmet or poorly met medical need and a large section of the population that is prepared to try anything without necessarily making any comment on the physical laws governing the universe.

    They just don’t care, like we do, and shouldn’t have to, either. It takes an enormous amount of specialized knowledge to reach the certainties that we have concerning what is likely to work better than placebo and what is not.

    If there are even only modest potential benefits being derived from all this this activity, then we are morally bound to take off the science lab coat and don our battered old physician’s top hat, and say, when it doesn’t really matter much for outcomes, “good luck to you, folks”. Shit, we are scarcely more than half a century away from wholesale use of unproven methods ourselves and we have nothing much to be smug about.

    I am NOT suggesting that the mainstream embarks wholesale upon the use of frank placebos. This is another area in which the skeptical mind can be somewhat one-track. That would not work.

Comments are closed.