Science-Based Medicine 101: Reproducibility

So far I have explained why most research (if not carefully designed) will lead to a false positive result. This inherent bias is responsible for many of the illusionary treatment benefits that we hear about so commonly through the media (whether they’re reporting about CAM or Western medicine), because it is their job to relay information in an entertaining way more so than an accurate manner (i.e. good science makes bad television).Then I explained a three step process for determining the trustworthiness of health news and research. We can remember these steps with a simple mnemonic: C-P-R.

The C stands for credibility- in other words, “consider the source” – is the research published in a top tier medical journal with a scientifically rigorous review process?

The P stands for plausibility- is the proposed finding consistent with known principles of physics, chemistry, and physiology or would accepting the result require us to suspend belief in everything we’ve learned about science to date?

And finally we arrive at R – reproducibility. If the research study were repeated, would similar results be obtained?

This third and final pillar of trustworthy science is a simple, but sometimes forgotten, principle. If there is a true cause and effect relationship observed by the researcher, then surely that cause and effect can be demonstrated again and again under the same conditions. Touching a hot stove burner always results in a burned hand. No matter how frequently you test this causal relationship, the result will be similar.

Sometimes conflicting results are obtained by repeating a study. When this happens, the reader should be careful in interpreting the conclusions – there may be a flaw in the study design, or it may be that the conclusions drawn were inaccurate. There could have been a false positive result, or no appreciable effect of the treatment under consideration, therefore leaving the results to chance. Flipping a coin gives you heads one minute and tails the next. Yet a person unfamiliar with coins could conclude (after one flip) that it has a head on both sides. In the end, therefore, one can be more confident in a study’s result if it is born out by other studies.

And so as we conclude this series, I hope that you now feel well equipped to perform CPR (credibility, plausibility, reproducibility checks) on health news. A little healthy skepticism can protect your brain from all the mixed health messages that barrage us each day. At the very least, now you’ll appreciate why most health news reports include an expert quote stating something to the effect of “it’s too early to know for sure if these findings are relevant.” That statement may be the most trustworthy of the entire report.

Posted in: Science and Medicine, Science and the Media

Leave a Comment (4) ↓

4 thoughts on “Science-Based Medicine 101: Reproducibility

  1. snfraser says:

    I think one critical component to reproducibility is that the results can be reproduced/replicated by others. So, replication needs to be independent. That is, by someone else like another researcher, or research group. I’m sure this is implicit in your post, but this point is really how the scientific system works. This is the thing that makes science so much better than other ways of knowing. Independent replication is a powerful “self-correcting” mechanism.

    Congrats on a nice series of posts.

  2. Ed Whitney says:

    When published research turns out to be false, it may come from a credible source and be plausible, but it still may be testing hypotheses that are improbable. John Ioannidis has written a lot about this problem, but his research is in gene-cancer relationships, where most genes are not related to cancer, but many initial “positive” relationships are due to chance.

    On one hand, a lot of sociological research tests plausible and probable hypotheses. There is good evidence, for example, that people with gobs and gobs of money are better off than people who don’t have any. You can get very low p values if you measure the number of airline trips they take, how many cars they own, and how big their houses are.

    On the other hand, really interesting research takes considerable risks by exploring hypotheses that have good theoretical reasons to be true, but (because of the complexity of biological systems) are likely to be false.

    There were great reasons to believe that vertebroplasty would relieve back pain from osteoporotic fractures. The sources that proposed this were credible. But osteoporotic back pain has many facets (so to speak). It is a complex system. It appears that there is little difference between vertebroplasty and a placebo procedure. This story is far from being over.

    So the reason that reproducibility is important is that research worth doing is done by credible, testing plausible hypotheses, that are still likely to be false, because living things are complicated. The effects of single variables are often lost amid the effect of other variables. But most well-done research is focused on a single variable. A plausible hypothesis may be tested and appear confirmed one time. The next time it is tested, some of those other variables may play a greater role in the study and it may fail to be confirmed.

    Reproducibility is important because good research takes risks. It tests plausible hypotheses that are likely to turn out in the long run to be at best only partially correct.

  3. yeahsurewhatever says:

    “This inherent bias is responsible for many of the illusionary treatment benefits that we hear about so commonly through the media (whether they’re reporting about CAM or Western medicine), because it is their job to relay information in an entertaining way more so than an accurate manner (i.e. good science makes bad television).”

    Have you read Steve Salerno’s “Journalist Bites Reality” article?

    I might have linked to it once in the distant past.

  4. I would like to hear thoughts on one area of medical science that seems to me that the burden of reproducibility is ignored – the occurrence of adverse reactions to pharmacological agents. We all know that adverse reactions to drugs are extremely difficult to predict and vary greatly from individual to individual. Some adverse reactions may occur in a very small percentage of those who take them, perhaps even less than one percent. And yet, despite these reactions varying so greatly, there is an acknowledgment that even extremely rare reactions can be real physiological phenomena not just a kind of bad placebo effect. In other words, patients aren’t told their side-effects must only be imagined since they don’t happen to everyone.

    Many drug reference books will describe a drugs “pharmacological action” in just a sentence or two and then list thirty or more “possible adverse reactions.” Have you ever thought about how odd it is that the same pharmacological action can cause such wildly variable and unpredictable physiological reactions? If we demand a high degree of reproducibility as means to discover truth, should we only acknowledge adverse reactions are real when they occur in a high percentage of subjects?

Comments are closed.