Articles

Persistence of Memory

I have steadily endeavored to keep my mind free so as to give up any hypothesis, however much beloved (and I cannot resist forming one on every subject), as soon as the facts are shown to be opposed to it.
— Charles R. Darwin

I’m getting old: 50, almost 51, and that’s over 350 in dog years. As a result of my advancing age there are things I do not get: tattoo’s, hip hop, visible undergarments, and those rectangular, square plastic glasses that seem so popular and look hideous on everyone. It gets harder to change.

I have been able to stick MD after my name for almost a quarter century now (175 dog years for those keeping track), and it does give a sense of perspective to the ebb and flow of medical therapies. Medicine for the last hundred years has been all about change. Dogma from last century is nonsense this century, all due to that damn science. It gets so tiresome having to learn something new.

Last month’s New England Journal of Medicine was another in a seemingly endless series of plus ça change, plus c’est la même chose moments.4 They published the results of the CORTICUS study, a trial that looked at the use of corticosteroids in septic shock. For background, septic shock occurs when a severe infection shuts down most important organs and the patient cannot maintain their blood pressure. Mortality is high and, depending on the patient, the infection and the infecting organism, mortality ranges from 30 to 110%. People can die several times during their admission and be temporarily brought back, at least from the sloppy definitions of the near death experience proponents (shameless plug for Quackcast 23). Sepsis is a Bad Disease. Patients in septic shock often have lower levels of cortisol in their blood than they should. So maybe giving replacement cortisol, in the form of hydrocortisone or prednisone, would decrease the death rates.

The results of the CORTICUS (I so want to be involved in a study that has an obscene acronym) suggests steroids are not effective. When compared to placebo, physiologic hydrocortisone (i.e. giving enough hydrocortisone to replace the deficit) was no better at preventing mortality. It didn’t work. Or didn’t it. Or did it? Which one?

I remember back when I was a resident back last century. We had to walk to the hospital barefoot in the snow, uphill, even in August. Made our own penicillin in the basement from moldy cheese. We took call every day and loved it. The sun was warmer, the sky bluer, water was wetter. Good times. It was also when the first study of sepsis and steroids was published. In that study they used relatively large doses and had an improved outcome.5

So everyone got steroids for sepsis.

Then another, better study, was published that demonstrated no change in mortality, but the patients who received steroids had more secondary infections. Bummer. More data. I have to change my practice. Damn. I hate it when that happens.

So the use of steroids faded. In medicine every therapy returns in a new way. I am waiting for the resurgence in the use of theophylline, for example. It’s going to happen.

Then it was noted that septic patients were relatively cortisol deficient. When tested their cortisol levels were lower than it should be in sepsis and that their adrenal function, the source of cortisol, was suppressed. Perhaps if patients received physiologic, replacement doses of steroids, they would do better.

Early studies looked promising, several small trials were done and several meta analyses suggested that there was a sweet spot for steroids, that indeed a lot was dangerous but a little was of benefit in decreasing mortality.

Although meta analyses are often subsequently disproved, it was compelling information. So steroids came back in vogue. Damn. Have to change practice again. Given that my mind is mostly filled with the lyrics of 1970’s rock music, it better not happen again. There is just so much I can learn.

Until the CORTICUS study. Now what. Do we give steroids? Or not? Or in a subgroup of patients? Damned if I know. My Critical Care colleagues tell me that they may yet be a role in patients with refractory sepsis since cortisone made them better faster, and generally the quicker you get out of a septic state the better you do (although not demonstrated in the CORTICUS study, it was not set up to determine outcome in this subgroup so the issue is still open to debate).

So over the last 25 years its yes and no and yes and mostly no when it comes to the treatment of sepsis.

This ebb and flow of medical knowledge is a standard feature of the practice of medicine and is, to the best of my knowledge, never a feature of sectarian or unscientific medicine (so-called Complementary and Alternative or Integrative Medicine).

I often wish I could practice unscientific medicine. Learn one all encompassing myth and never have to change again. CAM proponents often point to the changing interventions of medicine as a flaw. Look, they say, these guys don’t know what they are doing. One day its yes, next day its no, then its yes again. Do you know why a doctor’s mind is so clean and pure? Because she is always changing it.

Part of the problem is that in medicine we learn and practice evolves as the science behind the practice changes, and as an ever moving target, we are always refining practice. It is a strength of medicine, but leads to a degree of doubt and uncertainty and one thing people hate is doubt and uncertainty. Better a firm conviction about nonsense than a doubt about reality.

The other issue this study raises, and it is an important one when reading in clinical study, is sample size and its influence on the ability to determine differences in study groups. The smaller the study size, the more difficult it is to find a difference and the more likely it is that differences are due to noise. This concept is expressed as the power of a study.

The power of a study can be calculated in advance. In the accompanying editorial they note, given a 35% mortality rate, that to demonstrate a 15% decrease in relative risk of mortality would require 2600 patients. The CORTICUS study had 499 patients and would not be able to accurately determine small differences between the groups.

Note that to find a 15% difference, you need to study 2600 patients.

This is an issue in most clinical studies in both science-based medicine and in studies of sectarian modalities: the sample size is so small that “statistically significant” results are more likely due to background variation. If you have a small difference that is statistically significant between small groups, it is probably meaningless.

Look at the recent acupuncture and heartburn study:2 15 patients in each group. Even if the difference in the two groups were 100% and the end point was death i.e. 15 lived and 15 died, the results would still be meaningless, regardless of the statistical significance. Not enough patients.

Most studies, especially in the CAM literature, suffer from this issue. Most are simply extended case series. It’s the old joke.

One patient with a disease: “In my experience…”

Two patients with a disease: “In many patients…”

Three cases with a disease: “In case after case after case…”

BTW. Read Prior Probability: The Dirty Little Secret of “Evidence-Based Alternative Medicine” on this blog. Makes me feel like M.J. GUMBY: “Oh I have a piece of brain stuck in my head… oh my head hurts!” It’s what all statistics do to me. But it is a key concept in reading any medical literature, CAM or not.

Unfortunately, the problem with information, good, bad, or indifferent, is once it gets in the literature it stays. There are a lot of myths in the practice of medicine: normal temperature is 98.6, atelectasis causes fever, you have to double cover Pseudomonas, antibiotics are strong or powerful. The problem can be that with any strongly held opinion, it is hard to change your mind. Human psychology is such that people look for facts that confirm pre existing beliefs and ignore those that contradict them. Medicine is no different and it is interesting how misinformation persists when it is contradicted by good data.

The Journal of the American Medical Association published an article last month entitled “Persistence of Contradicted Claims in the Literature.”3

They looked at data that was initially based on observational epidemiology that were later contradicted by randomized clinical trials. In this case they looked at Vitamin E and cardiovascular benefits and Beta Carotene and Alzheimer’s.

Epidemiological studies suggested benefit from each, later clinical trials said, nope, sorry, we were mistaken.

It seems that despite excellent clinical trials, people cannot bring themselves to refute the older epidemiologic data and persist on citing the literature in support of these interventions that don’t work.

What is also interesting is the rationales for not believing the controlled trials, ranging from quibbles bout bias, differing populations, pharmacokinetics, confounding variables to poor endpoints.

So I suppose steroids will continue to be used in sepsis, depending on the patient and how the doc understands a complex and changing literature.

This article also brings some insight into all medicine, real or “alternative.” If you are convinced, for whatever reason, that you can treat a disease with water or needles or even, heaven forbid, steroids, it may be difficult to change your mind even if the contradicting data is good. It is how people are.

This brings home the issue as to how difficult it is to do convincing clinical trials: even the best trials can be unconvincing if you are unable to discount earlier, poorly designed or less powerful, studies. Bad trials, like those found in sectarian practices, crumble under the slightest touch. Once you read past the self aggrandizing abstract (most do not) you see that, like Oakland, there is no there, there.

In sectarian medicine, early trials, which tend to have positive results, are also the more poorly designed, biased trials. Once published, they persist in supporting unscientific modalities even after better trials demonstrate the lack of efficacy. The arc of publishing studies related to sectarian modalities is increasingly well done studies with decreasing efficacy until an excellent study shows no effect. But the early studies continue to pollute the intellectual waters. I wonder if there is such a thing as intellectual primacy: the first thing you learn, regardless of subsequent contradictory information, sticks better in the brain solely because it was the first learned.

It must be nice to be a CAM provider, or some MD’s I know: immune from ever having to learn or change.

Take home?

Medicine is hard, confusing, contradictory and always changing. Real docs may share more commonalities with CAM practitioners than they would like. Change is hard, especially for old fogey’s like me.

Makes we wish I were a homeopath.

Footnotes

  1. Sprung CL, Annane D, Keh D, Moreno R, Singer M, Freivogel K, Weiss YG, Benbenishty J, Kalenka A, Forst H, Laterre PF, Reinhart K, Cuthbertson BH, Payen D, Briegel J; CORTICUS Study Group. Hydrocortisone therapy for patients with septic shock. N Engl J Med. 2008 Jan 10;358(2):111-24. PMID: 18184957 Return to text
  2. Aliment Pharmacol Ther. 2007 Nov 15;26(10):1333-44. Epub 2007 Sep 17. Clinical trial: acupuncture vs. doubling the proton pump inhibitor dose in refractory heartburn.
    Dickman R, Schiff E, Holland A, Wright C, Sarela SR, Han B, Fass R. Return to text
  3. JAMA. 2007 Dec 5;298(21):2517-26. Persistence of contradicted claims in the literature. Tatsioni A, Bonitsis NG, Ioannidis JP. Return to text
  4. The more things change, the more things stay the same. Barbarian. Return to text
  5. A meta analysis: http://www.annals.org/cgi/content/full/141/1/47. A structured review: http://www.bmj.com/cgi/content/full/bmj;329/7464/480. Return to text

Posted in: Science and Medicine

Leave a Comment (14) ↓

14 thoughts on “Persistence of Memory

  1. apteryx says:

    You say:

    “Look at the recent acupuncture and heartburn study (2): 15 patients in each group. Even if the difference in the two groups were 100% and the end point was death i.e. 15 lived and 15 died, the results would still be meaningless, regardless of the statistical significance. Not enough patients.”

    Plenty of Phase I studies are no larger than that. If you did a Phase I study of a new cancer drug with 30 patients, and all 15 of the recipients were even still alive, much less feeling better, after all 15 of the standard care group had died, you would not only publish those results but would stampede to do Phase II trials as fast as you could. Of course, you would not claim that a study with so few people proved the efficacy of the new drug. But you would claim that it justified spending tens of millions of dollars on further research, and giving it to human subjects in that research. That’s not “meaningless.”

  2. David Gorski says:

    “Look at the recent acupuncture and heartburn study (2): 15 patients in each group. Even if the difference in the two groups were 100% and the end point was death i.e. 15 lived and 15 died, the results would still be meaningless, regardless of the statistical significance. Not enough patients.”

    Apteryx beat me to this. I’m sorry, Mark, but this is just plain bullshit. (Your presence on this blog is starting to have an effect on me, as evidenced by my disturbingly increasing tendency to use terms like “bullshit.” I hope you approve.)

    You’re totally neglecting something: The natural history of the disease. If you’re looking at a disease with in essence 100% mortality if untreated (unresectable or metastatic pancreatic cancer, for example), then doing a study with 30 patients, 15 in each group, in which there was 100% survival in the treated group and 100% mortality in the control group, it would be highly significant result. The odds of such a result by random chance would be as infitessimal as a 30C homeopathic remedy because about as close to all untreated pancreatic cancer patients as it is possible to imagine die without treatment. (The rare case reported who doesn’t die almost certainly never had pancreatic cancer in the first place.) Even patients whose disease is resected with completely negative margins can only expect at best a 30% chance of living five years.

    In fact, it would be so significant that patterns of practice might change overnight, mainly because there’s nothing else that we can offer pancreatic cancer patients that even comes close to this. It would be revolutionary. It would bring hope where at present there is none. In fact, because of the profound difference in survival observed in your example, in the case of something like pancreatic cacner it would be unethical to add more patients to the protocol. Treating future patients this way and observing continued 100% survival (or even 50% survival, assuming the first 15 were a fluke) would be adequate to validate the treatment as highly efficacious against pancreatic cancer, given that the natural history of metastatic pancreatic cancer is death within a matter of months.

    I know, I know. You’ll mention the Gonzalez protocol, the uncontrolled study with 12 patients who did better than historical controls with pancreatic cancer when treated with Gonzalez’s woo. However, that was not a controlled study prone to serious selection bias and all of the Gonzalez patients progressed; this was not a matter of survival versus death.

    No, Mark, the above statement remains, as you are so fond of calling various CAM propositions, bullshit.

    There, was that pugnacious enough for you? :-)

  3. DavidCT says:

    Can’t you imagine what a drag it would be to have no new things and ideas in your practice. I like the idea that there will always be more to learn even about things we think we know.

    For a thinking person being something like a homeopath would be hell on earth.

    Lots of luck Young fellow!

  4. caoimh says:

    “I wonder if there is such a thing as intellectual primacy: the first thing you learn, regardless of subsequent contradictory information, sticks better in the brain solely because it was the first learned.”

    The primacy effect??

    “The primacy effect, in psychology and sociology, is a cognitive bias that results from disproportionate salience of initial stimuli or observations. If, for example, a subject reads a sufficiently long list of words, he or she is more likely to remember words read toward the beginning than words read in the middle.”

  5. Scotty B says:

    “I have been able to stick MD after my name for almost a quarter century now (9175 dog years for those keeping track)”

    Um, you mean 175 dog years (not that I’m keeping track)?

    /Admittedly, I probably wouldn’t even have noticed if the 350 hadn’t been stuck in my head from the previous paragraph.

  6. I do think Mark overstated his point, because he didn’t clarify the application in proper context. The natural history of heartburn is variable – 30 patients will never be convincing. The natural history of pancreatic cancer is almost certain death. You can’t really compare these situations, but Mark did blur the lines and cause confusion by saying “even if the outcome were death.”

    Here’s another context – what if a homeopath claimed to cure 15 patients with pancreatic cancer with a purely homeopathic remedy? I wouldn’t buy it based upon one researcher and 15 patients. It is vastly more likely that the researcher was either profoundly in error or committed fraud. (This all gets back to the prior probability thing.) Such a claim would have to be replicated with absolute transparency – confirm that the diagnoses were correct, etc. Then it would be compelling.

    Everything depends on context.

  7. BlazingDragon says:

    A bit of hyperbole doesn’t take away from the main point of the post: Many doctors are unwilling to change “what they know works,” even in spite of evidence slapping them in the face. Of course, CAM practitioners are far worse, never changing what they do, even though their treatments are based on total BS and their “science” is plainly contradicted by reality. Their whole business model is never having to do expensive research, just claim it works and keep making money hand over fist.

    Medical doctors should be held to a much higher standard, but many resist changing their attitudes/treatments when faced with new (and better evidence). “If someone is presented with evidence that plainly contradicts their belief, they will often throw away the evidence and continue to believe.” This, as Dr. Crislip said, is part of human nature. There is no shame in allowing it to happen once in awhile, but there is a lot of shame in making the same mistake over and over again.

    My experience with doctors is that medicine is a lot harder in practice (compared to their expectations of driving a Mercedes when they were on the outside looking in as pre-med undergraduates) and many physicians resent how hard they have to work.

    Some of the complaints of doctors are legitimate (the way health plans continue to chip away at General Practitioners/Family doctors (despite those doctors seeing most of the patients) is nearly criminal in my book), but the awful truth about money in medicine is a separate issue from how medicine should be practiced, which does involve changing your practice when new evidence comes along. This particular behavior is something that most physicians have trouble with (speaking from a lot of personal experience).

    If I practiced organic synthesis the way I’ve seen most doctors practice medicine, I’d rarely make anything new, just keep repeating the same experiments that have failed or make stuff that other people have made many times before. The mental flexibility needed to practice medicine well is a difficult skill to acquire and keep. This requires keeping the ego well under control and, even then, requires some ego bruising once in awhile.

  8. David Gorski says:

    It’s a good thing we have our fearless leader around to calm the hotter and more sarcastic among his little stable of bloggers; he is correct in that perhaps I overreacted a bit, and I’m sorry about that. In my defense, let me just say that, whenever I read one of Mark’s posts, I can’t help but hear his voice reading it as though he’s doing one of his Quackcasts, in which he is more sarcastic and occasionally uses the term “bullshit.” This obviously had an effect on my impressionable mind.

    Steve’s also correct that the context is everything, which is what I was getting at in my rather crude manner. The problem, of course, is that Mark made a very confident claim without any qualifications or putting it in context, and that made it a very confident and wrong claim. That’s what got my dander up a bit as a cancer researcher. He also failed to take into account prior probability, a topic that has become one of the major themes of this blog.

    That being said, Steve is only partially correct. In the case of a disease with virtually a 100% mortality (metastatic pancreatic cancer), it is actually rather hard to produce survivors even with a screwed up protocol and selection bias because, well, a verified and well-documented report of a single long term survivor of metastatic pancreatic cancer could be published as a case report that the New England Journal of Medicine. It would be that unusual and significant. Consequently, even for a trial of something as incredibly improbable scientifically as homeopathy that produced such a result would have to be looked at. There are only a limited number of possible explanations for such an incredible result:

    1. Fraud

    2. Incompetence at clinical trials

    3. The patients surviving didn’t really have pancreatic cancer (which is really just a subset of #1 or #2). This is far more common in cancer trials of “CAM” therapies than you may think; tumors are often “diagnosed” using nonstandard and worthless methods and then “cured” with another set of nonstandard and worthless methods.

    4. The treatment worked. (Incredibly unlikely but the possibility must be considered.)

    Of course, #1, 2, and 3 should be verifiable after the fact, however, even in the case of an incredibly unlikely treatment such as homeopathy, it would be of dubious ethics to run a randomized trial after a result like the one described by Mark. After all, even if the original trial was incredibly unlikely to be correct from a prior probability and scientific standpoint, in light of such a study no IRB would approve a new randomized trial in which patients might be randomized to a 0% survival group when there is a report that the treatment group had 100% survival in a pilot study. It’s just not going to happen, and rightly so. However, because pancreatic cancer is so deadly, it would be perfectly valid to do a single-arm prospective (and, as Steve points out, totally transparent) study and use historical controls.

    The cancer clinical trials wonk in me just cannot be silenced.

    I should also point out that, that one statement aside, Mark’s post was otherwise excellent, as usual.

  9. SimonRX says:

    apteryx…

    “Plenty of Phase I studies are no larger than that. If you did a Phase I study of a new cancer drug with 30 patients, and all 15 of the recipients were even still alive, much less feeling better, after all 15 of the standard care group had died, you would not only publish those results but would stampede to do Phase II trials as fast as you could. Of course, you would not claim that a study with so few people proved the efficacy of the new drug. But you would claim that it justified spending tens of millions of dollars on further research, and giving it to human subjects in that research.”

    Just to clarify the phases of clinical trials:

    Phase I trials look at pharmacokinetic parameters in a small number of healthy volunteers. Phase II trials do pretty much the same things only in a small number of patients with the target disease. Both Phase I and II trials do not run long enough to see any effects on patients. The times that drug studies are stopped at this point if there are any serious adverse events that occur in the process (i.e. death from the drug). Phase III are the large clinical trials that involve placebos and are blinded and randomized. These are the ones that are powered enough to look for any clinical effect. Phase I or II trials are not designed or powered to see any efficacy of drug treatment. Therefore, the only results from these trials used to determine whether or not to continue testing the drug (and spending millions on it) are: its pharmacokinetics (its half life, bioavailability, clearance, concentrations in various tissues, etc…) and what side effects (death, organ toxicity, severe vomiting with each dose) the drug has.

  10. apteryx says:

    BlazingDragon writes:

    “Of course, CAM practitioners are far worse, never changing what they do, even though their treatments are based on total BS and their “science” is plainly contradicted by reality.”

    This may be true of homeopathy, but it is not true of all non-modern Western health care. You can see changes over time in what plants are used and how. If people observe that one febrifuge is more reliable than another, the use of the latter declines and becomes obsolete, while new practices similarly arise over time.

    Steven Novella writes:

    “Here’s another context – what if a homeopath claimed to cure 15 patients with pancreatic cancer with a purely homeopathic remedy? I wouldn’t buy it based upon one researcher and 15 patients. It is vastly more likely that the researcher was either profoundly in error or committed fraud. ”

    I would agree with that — although absent proof of fraud or diagnostic effor, if I had pancreatic cancer and the remedy were available, I’d sure use it, given that some chance of benefit outweighs, allegedly, zero chance of harm.

  11. pmoran says:

    Simonrx “Phase I trials look at pharmacokinetic parameters in a small number of healthy volunteers. Phase II trials do pretty much the same things only in a small number of patients with the target disease. Both Phase I and II trials do not run long enough to see any effects on patients. The times that drug studies are stopped at this point if there are any serious adverse events that occur in the process (i.e. death from the drug). Phase III are the large clinical trials that involve placebos and are blinded and randomized. These are the ones that are powered enough to look for any clinical effect. Phase I or II trials are not designed or powered to see any efficacy of drug treatment. ”

    This would apply to some pharamaceuticals. but not cancer treatments. It would be unethical to test chemotherapy on a normal person. Dose-finding should occur in patients with cancer, anyway.

    Phase ll studies usually do look for evidence of cancer remission. Phase lll studies would probably not occur unless there was already some evidence of clinical benefit, and they would usually compare the new treatment with best available care, not placebo (unless it was a symptomatic treatment such as painkiller).

  12. David Gorski says:

    This would apply to some pharamaceuticals. but not cancer treatments. It would be unethical to test chemotherapy on a normal person. Dose-finding should occur in patients with cancer, anyway.

    Indeed. Most patients in phase I oncology trials have advanced cancer. Although it is not the primary purpose of a phase I trial (dose escalation and identification of dose-limiting toxicities are), evidence of tumor regression or stabilization are sought and sometimes found.

  13. Mark Crislip says:

    1) Upon further reflection, yep. Bullshit. Bad example to make my point, that being: small numbers that have statistical significance may not have clinical significance.

    2) the 9 and the closed parenthesis are the same key and I hit them a the wrong time. Looks like Steve fixed it. But it goes to the difficult issue in medicine that you see what you expect to see, not what is there. A problem probably worthy of a post someday.

    3) I am definitely swimming with the big fish……

  14. pmoran says:

    Mark: “1) Upon further reflection, yep. Bullshit. Bad example to make my point, that being: small numbers that have statistical significance may not have clinical significance.”

    Is it worth changing this section, or further explaining what you meant? It could well be quoted elsewhere to reinforce the perception that a double standard is being applied to alternative studies.

    On further thought, I am not sure how to approach that, for a kind of double standard IS being applied. The threshold for general scientific acceptance of a treatment method is justifiably greater for methods that go against contrary evidence as to the nature of things, or which invoke processes or forces for which otherwise explainable clinical observations are the only evidence.

Comments are closed.