Articles

Persistence of Memory

I have steadily endeavored to keep my mind free so as to give up any hypothesis, however much beloved (and I cannot resist forming one on every subject), as soon as the facts are shown to be opposed to it.
— Charles R. Darwin

I’m getting old: 50, almost 51, and that’s over 350 in dog years. As a result of my advancing age there are things I do not get: tattoo’s, hip hop, visible undergarments, and those rectangular, square plastic glasses that seem so popular and look hideous on everyone. It gets harder to change.

I have been able to stick MD after my name for almost a quarter century now (175 dog years for those keeping track), and it does give a sense of perspective to the ebb and flow of medical therapies. Medicine for the last hundred years has been all about change. Dogma from last century is nonsense this century, all due to that damn science. It gets so tiresome having to learn something new.

Last month’s New England Journal of Medicine was another in a seemingly endless series of plus ça change, plus c’est la même chose moments.4 They published the results of the CORTICUS study, a trial that looked at Hydrocortisone therapy for patients with septic shock. N Engl J Med. 2008 Jan 10;358(2):111-24. PMID: 18184957 Return to text

  • Aliment Pharmacol Ther. 2007 Nov 15;26(10):1333-44. Epub 2007 Sep 17. Clinical trial: acupuncture vs. doubling the proton pump inhibitor dose in refractory heartburn.
    Dickman R, Schiff E, Holland A, Wright C, Sarela SR, Han B, Fass R. Return to text
  • JAMA. 2007 Dec 5;298(21):2517-26. Persistence of contradicted claims in the literature. Tatsioni A, Bonitsis NG, Ioannidis JP. Return to text
  • The more things change, the more things stay the same. Barbarian. Return to text
  • A meta analysis: http://www.annals.org/cgi/content/full/141/1/47. A structured review: http://www.bmj.com/cgi/content/full/bmj;329/7464/480. Return to text
  • Posted in: Science and Medicine

    Leave a Comment (14) ↓

    14 thoughts on “Persistence of Memory

    1. apteryx says:

      You say:

      “Look at the recent acupuncture and heartburn study (2): 15 patients in each group. Even if the difference in the two groups were 100% and the end point was death i.e. 15 lived and 15 died, the results would still be meaningless, regardless of the statistical significance. Not enough patients.”

      Plenty of Phase I studies are no larger than that. If you did a Phase I study of a new cancer drug with 30 patients, and all 15 of the recipients were even still alive, much less feeling better, after all 15 of the standard care group had died, you would not only publish those results but would stampede to do Phase II trials as fast as you could. Of course, you would not claim that a study with so few people proved the efficacy of the new drug. But you would claim that it justified spending tens of millions of dollars on further research, and giving it to human subjects in that research. That’s not “meaningless.”

    2. David Gorski says:

      “Look at the recent acupuncture and heartburn study (2): 15 patients in each group. Even if the difference in the two groups were 100% and the end point was death i.e. 15 lived and 15 died, the results would still be meaningless, regardless of the statistical significance. Not enough patients.”

      Apteryx beat me to this. I’m sorry, Mark, but this is just plain bullshit. (Your presence on this blog is starting to have an effect on me, as evidenced by my disturbingly increasing tendency to use terms like “bullshit.” I hope you approve.)

      You’re totally neglecting something: The natural history of the disease. If you’re looking at a disease with in essence 100% mortality if untreated (unresectable or metastatic pancreatic cancer, for example), then doing a study with 30 patients, 15 in each group, in which there was 100% survival in the treated group and 100% mortality in the control group, it would be highly significant result. The odds of such a result by random chance would be as infitessimal as a 30C homeopathic remedy because about as close to all untreated pancreatic cancer patients as it is possible to imagine die without treatment. (The rare case reported who doesn’t die almost certainly never had pancreatic cancer in the first place.) Even patients whose disease is resected with completely negative margins can only expect at best a 30% chance of living five years.

      In fact, it would be so significant that patterns of practice might change overnight, mainly because there’s nothing else that we can offer pancreatic cancer patients that even comes close to this. It would be revolutionary. It would bring hope where at present there is none. In fact, because of the profound difference in survival observed in your example, in the case of something like pancreatic cacner it would be unethical to add more patients to the protocol. Treating future patients this way and observing continued 100% survival (or even 50% survival, assuming the first 15 were a fluke) would be adequate to validate the treatment as highly efficacious against pancreatic cancer, given that the natural history of metastatic pancreatic cancer is death within a matter of months.

      I know, I know. You’ll mention the Gonzalez protocol, the uncontrolled study with 12 patients who did better than historical controls with pancreatic cancer when treated with Gonzalez’s woo. However, that was not a controlled study prone to serious selection bias and all of the Gonzalez patients progressed; this was not a matter of survival versus death.

      No, Mark, the above statement remains, as you are so fond of calling various CAM propositions, bullshit.

      There, was that pugnacious enough for you? :-)

    3. DavidCT says:

      Can’t you imagine what a drag it would be to have no new things and ideas in your practice. I like the idea that there will always be more to learn even about things we think we know.

      For a thinking person being something like a homeopath would be hell on earth.

      Lots of luck Young fellow!

    4. caoimh says:

      “I wonder if there is such a thing as intellectual primacy: the first thing you learn, regardless of subsequent contradictory information, sticks better in the brain solely because it was the first learned.”

      The primacy effect??

      “The primacy effect, in psychology and sociology, is a cognitive bias that results from disproportionate salience of initial stimuli or observations. If, for example, a subject reads a sufficiently long list of words, he or she is more likely to remember words read toward the beginning than words read in the middle.”

    5. Scotty B says:

      “I have been able to stick MD after my name for almost a quarter century now (9175 dog years for those keeping track)”

      Um, you mean 175 dog years (not that I’m keeping track)?

      /Admittedly, I probably wouldn’t even have noticed if the 350 hadn’t been stuck in my head from the previous paragraph.

    6. I do think Mark overstated his point, because he didn’t clarify the application in proper context. The natural history of heartburn is variable – 30 patients will never be convincing. The natural history of pancreatic cancer is almost certain death. You can’t really compare these situations, but Mark did blur the lines and cause confusion by saying “even if the outcome were death.”

      Here’s another context – what if a homeopath claimed to cure 15 patients with pancreatic cancer with a purely homeopathic remedy? I wouldn’t buy it based upon one researcher and 15 patients. It is vastly more likely that the researcher was either profoundly in error or committed fraud. (This all gets back to the prior probability thing.) Such a claim would have to be replicated with absolute transparency – confirm that the diagnoses were correct, etc. Then it would be compelling.

      Everything depends on context.

    7. BlazingDragon says:

      A bit of hyperbole doesn’t take away from the main point of the post: Many doctors are unwilling to change “what they know works,” even in spite of evidence slapping them in the face. Of course, CAM practitioners are far worse, never changing what they do, even though their treatments are based on total BS and their “science” is plainly contradicted by reality. Their whole business model is never having to do expensive research, just claim it works and keep making money hand over fist.

      Medical doctors should be held to a much higher standard, but many resist changing their attitudes/treatments when faced with new (and better evidence). “If someone is presented with evidence that plainly contradicts their belief, they will often throw away the evidence and continue to believe.” This, as Dr. Crislip said, is part of human nature. There is no shame in allowing it to happen once in awhile, but there is a lot of shame in making the same mistake over and over again.

      My experience with doctors is that medicine is a lot harder in practice (compared to their expectations of driving a Mercedes when they were on the outside looking in as pre-med undergraduates) and many physicians resent how hard they have to work.

      Some of the complaints of doctors are legitimate (the way health plans continue to chip away at General Practitioners/Family doctors (despite those doctors seeing most of the patients) is nearly criminal in my book), but the awful truth about money in medicine is a separate issue from how medicine should be practiced, which does involve changing your practice when new evidence comes along. This particular behavior is something that most physicians have trouble with (speaking from a lot of personal experience).

      If I practiced organic synthesis the way I’ve seen most doctors practice medicine, I’d rarely make anything new, just keep repeating the same experiments that have failed or make stuff that other people have made many times before. The mental flexibility needed to practice medicine well is a difficult skill to acquire and keep. This requires keeping the ego well under control and, even then, requires some ego bruising once in awhile.

    8. David Gorski says:

      It’s a good thing we have our fearless leader around to calm the hotter and more sarcastic among his little stable of bloggers; he is correct in that perhaps I overreacted a bit, and I’m sorry about that. In my defense, let me just say that, whenever I read one of Mark’s posts, I can’t help but hear his voice reading it as though he’s doing one of his Quackcasts, in which he is more sarcastic and occasionally uses the term “bullshit.” This obviously had an effect on my impressionable mind.

      Steve’s also correct that the context is everything, which is what I was getting at in my rather crude manner. The problem, of course, is that Mark made a very confident claim without any qualifications or putting it in context, and that made it a very confident and wrong claim. That’s what got my dander up a bit as a cancer researcher. He also failed to take into account prior probability, a topic that has become one of the major themes of this blog.

      That being said, Steve is only partially correct. In the case of a disease with virtually a 100% mortality (metastatic pancreatic cancer), it is actually rather hard to produce survivors even with a screwed up protocol and selection bias because, well, a verified and well-documented report of a single long term survivor of metastatic pancreatic cancer could be published as a case report that the New England Journal of Medicine. It would be that unusual and significant. Consequently, even for a trial of something as incredibly improbable scientifically as homeopathy that produced such a result would have to be looked at. There are only a limited number of possible explanations for such an incredible result:

      1. Fraud

      2. Incompetence at clinical trials

      3. The patients surviving didn’t really have pancreatic cancer (which is really just a subset of #1 or #2). This is far more common in cancer trials of “CAM” therapies than you may think; tumors are often “diagnosed” using nonstandard and worthless methods and then “cured” with another set of nonstandard and worthless methods.

      4. The treatment worked. (Incredibly unlikely but the possibility must be considered.)

      Of course, #1, 2, and 3 should be verifiable after the fact, however, even in the case of an incredibly unlikely treatment such as homeopathy, it would be of dubious ethics to run a randomized trial after a result like the one described by Mark. After all, even if the original trial was incredibly unlikely to be correct from a prior probability and scientific standpoint, in light of such a study no IRB would approve a new randomized trial in which patients might be randomized to a 0% survival group when there is a report that the treatment group had 100% survival in a pilot study. It’s just not going to happen, and rightly so. However, because pancreatic cancer is so deadly, it would be perfectly valid to do a single-arm prospective (and, as Steve points out, totally transparent) study and use historical controls.

      The cancer clinical trials wonk in me just cannot be silenced.

      I should also point out that, that one statement aside, Mark’s post was otherwise excellent, as usual.

    9. SimonRX says:

      apteryx…

      “Plenty of Phase I studies are no larger than that. If you did a Phase I study of a new cancer drug with 30 patients, and all 15 of the recipients were even still alive, much less feeling better, after all 15 of the standard care group had died, you would not only publish those results but would stampede to do Phase II trials as fast as you could. Of course, you would not claim that a study with so few people proved the efficacy of the new drug. But you would claim that it justified spending tens of millions of dollars on further research, and giving it to human subjects in that research.”

      Just to clarify the phases of clinical trials:

      Phase I trials look at pharmacokinetic parameters in a small number of healthy volunteers. Phase II trials do pretty much the same things only in a small number of patients with the target disease. Both Phase I and II trials do not run long enough to see any effects on patients. The times that drug studies are stopped at this point if there are any serious adverse events that occur in the process (i.e. death from the drug). Phase III are the large clinical trials that involve placebos and are blinded and randomized. These are the ones that are powered enough to look for any clinical effect. Phase I or II trials are not designed or powered to see any efficacy of drug treatment. Therefore, the only results from these trials used to determine whether or not to continue testing the drug (and spending millions on it) are: its pharmacokinetics (its half life, bioavailability, clearance, concentrations in various tissues, etc…) and what side effects (death, organ toxicity, severe vomiting with each dose) the drug has.

    10. apteryx says:

      BlazingDragon writes:

      “Of course, CAM practitioners are far worse, never changing what they do, even though their treatments are based on total BS and their “science” is plainly contradicted by reality.”

      This may be true of homeopathy, but it is not true of all non-modern Western health care. You can see changes over time in what plants are used and how. If people observe that one febrifuge is more reliable than another, the use of the latter declines and becomes obsolete, while new practices similarly arise over time.

      Steven Novella writes:

      “Here’s another context – what if a homeopath claimed to cure 15 patients with pancreatic cancer with a purely homeopathic remedy? I wouldn’t buy it based upon one researcher and 15 patients. It is vastly more likely that the researcher was either profoundly in error or committed fraud. ”

      I would agree with that — although absent proof of fraud or diagnostic effor, if I had pancreatic cancer and the remedy were available, I’d sure use it, given that some chance of benefit outweighs, allegedly, zero chance of harm.

    11. pmoran says:

      Simonrx “Phase I trials look at pharmacokinetic parameters in a small number of healthy volunteers. Phase II trials do pretty much the same things only in a small number of patients with the target disease. Both Phase I and II trials do not run long enough to see any effects on patients. The times that drug studies are stopped at this point if there are any serious adverse events that occur in the process (i.e. death from the drug). Phase III are the large clinical trials that involve placebos and are blinded and randomized. These are the ones that are powered enough to look for any clinical effect. Phase I or II trials are not designed or powered to see any efficacy of drug treatment. ”

      This would apply to some pharamaceuticals. but not cancer treatments. It would be unethical to test chemotherapy on a normal person. Dose-finding should occur in patients with cancer, anyway.

      Phase ll studies usually do look for evidence of cancer remission. Phase lll studies would probably not occur unless there was already some evidence of clinical benefit, and they would usually compare the new treatment with best available care, not placebo (unless it was a symptomatic treatment such as painkiller).

    12. David Gorski says:

      This would apply to some pharamaceuticals. but not cancer treatments. It would be unethical to test chemotherapy on a normal person. Dose-finding should occur in patients with cancer, anyway.

      Indeed. Most patients in phase I oncology trials have advanced cancer. Although it is not the primary purpose of a phase I trial (dose escalation and identification of dose-limiting toxicities are), evidence of tumor regression or stabilization are sought and sometimes found.

    13. Mark Crislip says:

      1) Upon further reflection, yep. Bullshit. Bad example to make my point, that being: small numbers that have statistical significance may not have clinical significance.

      2) the 9 and the closed parenthesis are the same key and I hit them a the wrong time. Looks like Steve fixed it. But it goes to the difficult issue in medicine that you see what you expect to see, not what is there. A problem probably worthy of a post someday.

      3) I am definitely swimming with the big fish……

    14. pmoran says:

      Mark: “1) Upon further reflection, yep. Bullshit. Bad example to make my point, that being: small numbers that have statistical significance may not have clinical significance.”

      Is it worth changing this section, or further explaining what you meant? It could well be quoted elsewhere to reinforce the perception that a double standard is being applied to alternative studies.

      On further thought, I am not sure how to approach that, for a kind of double standard IS being applied. The threshold for general scientific acceptance of a treatment method is justifiably greater for methods that go against contrary evidence as to the nature of things, or which invoke processes or forces for which otherwise explainable clinical observations are the only evidence.

    Comments are closed.