Articles

Of SBM and EBM Redux. Part I: Does EBM Undervalue Basic Science and Overvalue RCTs?

During the most recent kerfuffle about whether or not Evidence-Based Medicine can legitimately claim to be science-based medicine, it became clear to me that a whole, new round of discussion and documentation is necessary. This is frustrating because I’ve already done it several times, most recently less than a year ago. Moreover, I’ve provided a table of links to the whole series at the bottom of each post*…Never mind, here goes, and I hope this will be the last time it is necessary because I’ll try to make this the “go to” series of posts for any future such confusions.

The points made in this series, most of which link to posts in which I originally made them, are in response to arguments from statistician Steve Simon, whose essay, Is there something better than Evidence Based Medicine out there?, was the topic of Dr. Gorski’s rebuttal on Monday of this week, and also from several of the comments following that rebuttal. Mr. Simon has since revised his original essay to an extent, which I’ll take into account. I’ll frame this as a series of assertions by those who doubt that EBM is deficient in the ways that we at SBM have argued, followed in each case by my response.

First, a disclaimer: I don’t mean to gang up on Mr. Simon personally; others hold opinions similar to his, but his essay just happens to be a convenient starting point for this discussion. FWIW, prior to this week I perused a bit of his blog, after having read one of his comments here, and found it to be well written and informative.

 What’s in a Name?

One of Mr. Simon’s objections, in his revision, is this:

What is SBM? Here’s a definition found on the opening entry in the SBM blog:

“the use of the best scientific evidence available, in the light of our cumulative scientific knowledge from all relevant disciplines, in evaluating health claims, practices, and products.” http://www.sciencebasedmedicine.org/?p=1

But how does this differ from David Sackett’s definition of EBM?

“the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.” http://www.bmj.com/content/312/7023/71.full

The only substantial difference I see is the adjective “scientific” that appears twice in the definition of SBM. The claim on the SBM blog is that EBM ignores scientific plausibility. Actually, ignores is too strong a word.

“EBM ‘levels of evidence’ hierarchy renders each entry sufficient to trump those below it. Thus a ‘positive’ clinical trial is given more weight than ‘physiology, bench research or “first principles”,’ even when the latter definitively refute the claim.”  http://www.sciencebasedmedicine.org/?p=42

(I agree that “ignore” is too strong a word, but I didn’t actually write it that way, as Dr. Gorski pointed out and as I think Mr. Simon was acknowledging above.)

A difference between Sackett’s definition and ours is that by “current best evidence” Sackett means the results of RCTs. I realize that this assertion requires documentation, which will come below. A related issue is the definition of “science.” In common use the word has at least three, distinct meanings: 1. The scientific pursuit, including the collective institutions and individuals who “do” science; 2. The scientific method; 3. The body of knowledge that has emerged from that pursuit and method (I’ve called this “established knowledge”; Dr. Gorski has called it “settled science”).

I will argue that when EBM practitioners use the word “science,” they are overwhelmingly referring to a small subset of the second definition: RCTs conceived and interpreted by frequentist statistics. We at SBM use “science” to mean both definitions 2 and 3, as the phrase “cumulative scientific knowledge from all relevant disciplines” should make clear. That is the important distinction between SBM and EBM. “Settled science” refutes many highly implausible medical claims—that’s why they can be judged highly implausible. EBM, as we’ve shown and will show again here, mostly fails to acknowledge this fact.

Finally, Mr. Simon has misinterpreted our goal at SBM:

But if someone wants to point out that EBM needs work, I’m fine with that. I dislike that they think that EBM needs to be replaced with something better.

You see EBM as being wrong often enough that you see value in creating a new label, SBM. I see SBM as being that portion of EBM that is being done thoughtfully and carefully, and don’t see the need for a new label.

I generally bristle when people want to create a new and improved version of EBM and then give it a new label.

I am as harshly critical of the hierarchy of evidence as anyone. I see this as something that will self-correct over time, and I see people within EBM working both formally and informally to replace the rigid hierarchy with something that places each research study in context. I’m staying with EBM because I believe that people who practice EBM thoughtfully do consider mechanisms carefully. That includes the Cochrane Collaboration.

Mr. Simon, we agree! Yes, we are pointing out that EBM needs work. Yes, SBM is that (tiny) portion of EBM that is being done thoughtfully and carefully, and if it were mainly done that way there would be no need to call attention to the point. Our goal is not to change the name of EBM (“give it a new label”). Our goal is to convince EBM to live up to its current name. Yes, it may self-correct over time, but we are trying to shorten that time. Bad things have unnecessarily happened, in part due to EBM’s scientific blind spot: As currently practiced, it doesn’t rationally consider all the evidence. We don’t see much evidence that people at the highest levels of EBM, eg, Sackett’s Center for EBM or Cochrane, are “working both formally and informally to replace the rigid hierarchy with something that places each research study in context.”

We chose to call our blog “science-based medicine” only because the term “evidence-based medicine” had already been taken, and we needed to distinguish ourselves from the inaccurate use of the word “evidence” in “EBM.” I’ve written about this before, and have made the point utterly clear:

These are the reasons that we call our blog “Science-Based Medicine.” It is not that we are opposed to EBM, nor is it that we believe EBM and SBM to be mutually exclusive. On the contrary: EBM is currently a subset of SBM, because EBM by itself is incomplete. We eagerly await the time that EBM considers all the evidence and will have finally earned its name. When that happens, the two terms will be interchangeable.

Plausibility Misinterpreted

Mr. Simon’s interpretation of our view of plausibility, like that of many others, is wrong:

I would argue further that it is a form of methodolatry to insist on a plausible scientific mechanism as a pre-requisite for ANY research for a medical intervention. It should be a strong consideration, but we need to remember that many medical discoveries preceded the identification of a plausible scientific mechanism.

I think, from his revision, that Mr. Simon understood Dr. Gorski’s explanation of why this was wrong, but I’m not certain. The misrepresentation of scientific plausibility is an issue that I’ve faced for years, as explained previously here:

Plausibility ≠ Knowing the Mechanism

Let’s first dispense with a simple misunderstanding: We, by which I mean We Supreme Arbiters of Plausibility (We SAPs) here at SBM, do not require knowing the mechanism of some putative effect in order to deem it plausible. This seems so obvious that it ought not be necessary to repeat it over and over again, and yet the topic can’t be broached without some nebbishy South Park do-gooder chanting a litany of “just because you don’t know how it works doesn’t mean it can’t work,” as if that were a compelling or even relevant rebuttal. Let’s get this straight once and for all: IT ISN’T.

Steve Novella explained why at the Yale conference and again here. We talked about it at TAM7 last summer. For a particularly annoying example, read the three paragraphs beginning with “Mr. Gagnier’s understanding of biological plausibility” here.

OK, I’ll admit that I’m beginning to learn something from such frustration. Perhaps we’ve not been so good at explaining what we mean by plausibility. The point is not that we don’t know a particular mechanism for homeopathy, for example; the point is that any proposed mechanism would necessarily violate scientific principles that rest on far more solid ground than any number of equivocal, bias-and-error-prone clinical trials could hope to overturn. The same is true for “energy medicine” and for claims based on non-existent anatomical structures (iridology, reflexology, auricular acupuncture, meridians, chiropractic “subluxations”), non-existent physiologic functions (“craniosacral rhythms“), or non-existent anatomic-physiologic relations (“neurocranial restructuring,” “detoxification” with coffee enemas, dissolving tumors with orally administered pancreatic enzymes). The spectrum of implausible health claims euphemistically dubbed “CAM” is full of such nonsense.

Reader daedalus2u proposed a useful way to clarify the point:

I think the idea of prior plausibility should actually be reframed into one of a lack of prior implausibility. It isn’t that one should have reasons to positively think that something is plausible before testing it, but rather that one should not be able to come up with reasons (actually data) why it is fatally implausible.

Some of what We deem implausible will not be fatally so, of course. Implausibility can be based not only on established physical and biological knowledge, but also on studies, as is the case for sticking needles into people, injecting them with chelating agents, or claiming that autism is caused by childhood immunizations.

EBM, Basic Science, and RCTs

Steve Simon wrote, “I have not seen any serious evidence of EBM relying exclusively on RCTs. That’s certainly not what David Sackett was proposing in the 1996 BMJ editorial…” And: “No thoughtful practitioner of EBM, to my knowledge, has suggested that EBM ignore scientific mechanisms.”

Want serious evidence? Consider these quotations from Cochrane reviews, originally posted here:

In view of the absence of evidence it is not possible to comment on the use of homeopathy in treating dementia.

There is not enough evidence to reliably assess the possible role of homeopathy in asthma. As well as randomised trials, there is a need for observational data to document the different methods of homeopathic prescribing and how patients respond.

There is currently little evidence for the efficacy of homeopathy for the treatment of ADHD. Development of optimal treatment protocols is recommended prior to further randomised controlled trials being undertaken.

Though promising, the data were not strong enough to make a general recommendation to use Oscillococcinum for first-line treatment of influenza and influenza-like syndromes. Further research is warranted but the required sample sizes are large.

Yes, EBM undervalues basic science and overvalues RCTs when the former is sufficient to reject a claim. EBM also undervalues experimental evidence other than RCTs when the former is sufficient to reject a claim, as will be discussed. Here is how a truly evidence-based review might conclude a discussion of homeopathy for dementia:

The probability that homeopathy is specifically therapeutic for dementia is, for all practical purposes, zero.

The following is from my first post on the topic, in which I reviewed the overwhelming evidence—from basic science and pre-clinical research—that homeopathic ‘remedies’ have no, specific therapeutic actions, and wondered why the most esteemed exponents of EBM have written that such treatments are “promising” and that “further randomized trials are needed.” I included the Center for Evidence-based Medicine’s formal “Levels of Evidence” scheme (not copied here), the pertinent quotation from Sackett’s 1996 editorial, my opinion that this failure of EBM was initially unintended, how Sackett et al eventually did address “CAM,” and the Cochrane abstracts quoted above:

It wasn’t meant to be like this. When I first discussed with my fellow bloggers the curious absence of established knowledge in the EBM “levels of evidence” hierarchy, at least one insisted that this could not be true, and in a sense he was correct. David Sackett and other innovators of EBM do include basic science in their discussions, but they recommend invoking it only when there are no clinical trials to consider:

Evidence based medicine is not restricted to randomised trials and meta-analyses. It involves tracking down the best external evidence with which to answer our clinical questions…And sometimes the evidence we need will come from the basic sciences such as genetics or immunology. It is when asking questions about therapy that we should try to avoid the non-experimental approaches, since these routinely lead to false positive conclusions about efficacy. Because the randomised trial, and especially the systematic review of several randomised trials, is so much more likely to inform us and so much less likely to mislead us, it has become the “gold standard” for judging whether a treatment does more good than harm.

That statement is consistent with EBM’s formal relegation of established knowledge to “level 5,” as seen in the Figure. I am not a historian of EBM and don’t care to be, but I suspect that the explanation for this choice is that “they never saw ‘CAM’ coming.” In other words, it probably didn’t occur to Sackett and other EBM pioneers that anyone would consider performing clinical trials of methods that couldn’t pass the muster of scientific plausibility. Their primary concern was to emphasize the insufficiency of basic science evidence in determining the safety and effectiveness of new treatments. In that they were quite correct, but trials of “CAM” have since reminded us that although established knowledge may be an insufficient basis for accepting a treatment claim, it is still a necessary one.

Take note: Sackett wrote, “we should try to avoid the non-experimental approaches, since these routinely lead to false positive conclusions about efficacy.” My point is that pre-RCT evidence does not routinely (if ever) lead to false negative conclusions. In that passage, moreover, Sackett seems to suggest that the only alternative to a “non-experimental approach” is an RCT; yet there are often other types of experiments that can definitively refute treatment claims, as will be discussed. Eventually Sackett et al did catch wind of “CAM,” but they got it exactly wrong:

Lacking that perspective, Sackett’s Center for Evidence-Based Medicine promulgates an “Introduction to evidence-based complementary medicine” by “CAM” researcher Andrew Vickers. There is not a mention of established knowledge in it, although there are references to several claims, including homeopathy, that are refuted by things that we already know. Vickers is also on the advisory board of the Cochrane CAM Field, along with Wayne Jonas and several other “CAM” enthusiasts.

In another post I cited the 2006 Cochrane Review of Laetrile:

A 2006 Cochrane Review of Laetrile for cancer would, if its recommendations were realized, stand the rationale for RCTs on its head:

The most informative way to understand whether Laetrile is of any use in the treatment of cancer, is to review clinical trials and scientific publications. Unfortunately no studies were found that met the inclusion criteria for this review.

Authors’ conclusions

The claim that Laetrile has beneficial effects for cancer patients is not supported by data from controlled clinical trials. This systematic review has clearly identified the need for randomised or controlled clinical trials assessing the effectiveness of Laetrile or amygdalin for cancer treatment.

Why does this stand the rationale for RCTs on its head? A definitive case series led by the Mayo Clinic in the early 1980s had overwhelmingly demonstrated, to the satisfaction of all reasonable physicians and biomedical scientists, that not only were the therapeutic claims for Laetrile baseless, but that the substance is dangerous. The subjects did so poorly that there would have been no room for a meaningful advantage in outcome with active treatment compared to placebo or standard treatment… The Mayo case series “closed the book on Laetrile,” the most expensive health fraud in American history at the time, only to have it reopened more than 20 years later by well-meaning Cochrane reviewers who seemed oblivious of the point of an RCT.

Is that review not serious evidence that the Cochrane Collaboration overvalues RCTs? In this case, moreover, it wasn’t only basic science that Cochrane ignored, but a definitive piece of clinical research that was not an RCT. Sure, I know that Cochrane is not the only pinnacle of EBM, but it’s one of them.

In both that post and another, I called attention to a statement that Edzard Ernst, the most prolific EBM-style “CAM” researcher of the past 20 years, had made in 2003:

A couple of years ago I was surprised to find that one of the authors of [the Cochrane Laetrile] review was Edzard Ernst, a high-powered academic who over the years has undergone a welcomed transition from cautious supporter to vocal critic of much “CAM” research and many “CAM” methods. He is now a valuable member of our new organization, the Institute for Science in Medicine, and we are very happy to have him. I believe that his belated conversion to healthy skepticism was due, in large part, to his allegiance to the formal tenets of EBM. I recommend a short debate published in 2003 in Dr. Ernst’s Focus on Alternative and Complementary Therapies (FACT), pitting Jacqueline’s countryman Cees Renckens against Dr. Ernst himself. Dr. Ernst responded to Dr. Renckens’s plea to apply science to “CAM” claims with this statement:

In the context of EBM, a priori plausibility has become less and less important. The aim of EBM is to establish whether a treatment works, not how it works or how plausible it is that it may work. The main tool for finding out is the RCT. It is obvious that the principles of EBM and those of a priori plausibility can, at times, clash, and they often clash spectacularly in the realm of CAM.

I’ve discussed that debate before on SBM, and I consider it exemplary of what is wrong with how EBM weighs the import of prior probability. Dr. Ernst, if you are reading this, I’d be interested to know whether your views have changed. I hope that you no longer believe that human subjects ought to be submitted to a randomized, controlled trial of Laetrile!

Uh, talk about “suggesting that EBM ignore scientific mechanisms”! When the principles of EBM and those of a priori plausibility clash spectacularly in the realm of CAM, it is a priori plausibility that should take precedence—not merely because the latter renders RCTs unnecessary, but because for such questions RCTs tend to confuse rather than clarify, as will be discussed further in the next part of this series.

I am happy to report that Dr. Ernst wrote me privately about that passage, with the answer that I’d mostly hoped for:

Have I changed my mind? I am not as sure as the sceptics seem to be that I ever was a supporter of CAM and I am still a bit sceptic about the sceptics [which perhaps makes me the "ueber-sceptic"]. Would I argue for more Laetrile studies? NO.

Even more to the point, perhaps, is a recent editorial by Dr. Ernst in which he calls homeopathy “absurd” and compares it to other, obvious absurdities, which I doubt he’d have done only a few years ago:

Should we keep an open mind about astrology, perpetual motion, alchemy, alien abduction, and sightings of Elvis Presley? No, and we are happy to confess that our minds have closed down on homeopathy in the same way.

This kind of clear thinking, as easy as it ought to be for intelligent people, seems oddly difficult for those steeped in EBM. I’ll offer another example in part 2, as part of my response to Mr. Simon’s assertion that “There is some societal value in testing therapies that are in wide use, even though there is no scientifically valid reason to believe that those therapies work.”

*The Prior Probability, Bayesian vs. Frequentist Inference, and EBM Series:

1. Homeopathy and Evidence-Based Medicine: Back to the Future Part V

2. Prior Probability: The Dirty Little Secret of “Evidence-Based Alternative Medicine”

3. Prior Probability: the Dirty Little Secret of “Evidence-Based Alternative Medicine”—Continued

4. Prior Probability: the Dirty Little Secret of “Evidence-Based Alternative Medicine”—Continued Again

5. Yes, Jacqueline: EBM ought to be Synonymous with SBM

6. The 2nd Yale Research Symposium on Complementary and Integrative Medicine. Part II

7. H. Pylori, Plausibility, and Greek Tragedy: the Quirky Case of Dr. John Lykoudis

8. Evidence-Based Medicine, Human Studies Ethics, and the ‘Gonzalez Regimen’: a Disappointing Editorial in the Journal of Clinical Oncology Part 1

9. Evidence-Based Medicine, Human Studies Ethics, and the ‘Gonzalez Regimen’: a Disappointing Editorial in the Journal of Clinical Oncology Part 2

10. Of SBM and EBM Redux. Part I: Does EBM Undervalue Basic Science and Overvalue RCTs?

11. Of SBM and EBM Redux. Part II: Is it a Good Idea to test Highly Implausible Health Claims?

12. Of SBM and EBM Redux. Part III: Parapsychology is the Role Model for “CAM” Research

13. Of SBM and EBM Redux. Part IV: More Cochrane and a little Bayes

14. Of SBM and EBM Redux. Part IV, Continued: More Cochrane and a little Bayes

15. Cochrane is Starting to ‘Get’ SBM!

16. What is Science? 

Posted in: Basic Science, Medical Academia, Science and Medicine

Leave a Comment (142) ↓

142 thoughts on “Of SBM and EBM Redux. Part I: Does EBM Undervalue Basic Science and Overvalue RCTs?

  1. BillyJoe says:

    Stolen from another of Kimball Atwood’s posts:

    Although a reasonably high prior probability may not be a SUFFICIENT basis for incorporating a treatment into general use, it is a NECESSARY one.

    Moreover, high prior probability is a NECESSARY basis for seriously considering such a treatment at all; that is, for both scientific and ethical reasons it is a PREREQUISITE for performing a randomized, controlled human trial.

    A perfect and succinct summary I think.

  2. mark says:

    Does it need to be more explicitly stated that you are saying RCTs are overvalued only when the results of preclinical science suggest low, or zero pre-test probability? (Apologies if that is not what you are saying) Otherwise, isn’t there a danger that quacks can claim that their brand of snake oil is good for health based on pre-clinical science even if it is not supported by RCTs or other clinical evidence, and justify this by saying that RCTs are overvalued, and that they are practising SBM?

    I think it would be helpful to add a category “what is SBM” or something similar under your list at the top of the web page with this and related posts. You might have written about this before repeatedly, but it is not that easy to find on the website.

  3. Does it need to be more explicitly stated that you are saying RCTs are overvalued only when the results of preclinical science suggest low, or zero pre-test probability? (Apologies if that is not what you are saying)

    Yes, that is what I’m saying, have said repeatedly, and I don’t know how much more explicit I can get (see Billy Joe’s comment). Please read, in its entirety,

    Yes, Jacqueline: EBM ought to be Synonymous with SBM

    Regarding “what is SBM,” look here: http://www.sciencebasedmedicine.org/?p=1

  4. David Gorski says:

    My point is that pre-RCT evidence does not routinely (if ever) lead to false negative conclusions.

    Excellent point. Plausibility based on pre-clinical data fairly often results in suggested therapies that fail to pan out when tested in RCTs. In other words, preclinical plausibility often leads to false positives, but it rarely leads to false negatives.

  5. Jann Bellamy says:

    ” . . . isn’t there a danger that quacks can claim that their brand of snake oil is good for health based on pre-clinical science . . .”

    They already do this, but under investigation their “theories” fall apart. The chiropractic “subluxation” and homeopathy are two examples.

  6. evilrobotxoxo says:

    Re: the idea that preclinical data rarely lead to false negatives. I agree with this, but I don’t think that anyone can really back this assertion up with numbers. The reason why is because if something tests negative in a pre-clinical setting, it will never be tested clinically, so there’s a bias present there.

    A second point is that another way of framing the problem with EBM is that the hierarchy places small, low-quality RCTs above a massive, excellent-quality body of pre-clinical studies. In principle, I actually don’t have a problem with the idea that RCTs should be at the top of the hierarchy. If enough high-quality RCTs showed that homeopathy had a strong effect, then that really would mean that all of chemistry and physics were wrong. The problem is the idea that ANY RCT data trumps ANY basic science data, regardless of quality.

    In a way, the problem is that EBM is like the internet, which was designed with the idea that everyone would play according to the rules, so there was no security built in, and it had to be added on later. EBM assumes that the only reason a clinical trial would be performed is because certain standards were met in terms of basic science data, case reports, unblinded studies, etc. The people who run RCTs for non-science-based “therapies” that don’t meet those standards are like spammers on the internet, who take advantage of holes in the system.

  7. @ evilrobotxoxo:

    I figured someone would raise that point. It presages Part II of this series, but in the meantime consider two scenarios. In the first, the claim is so unlikely, based on established knowledge, that no practical amount of clinical data can save it. Homeopathy, claims based on psychokinesis (eg, “distant healing,” “therapeutic touch,” “applied kinesiology”), claims based on other fanciful “energies” (craniosacral rhythms) or claims based on fanciful homunculi (iridology, reflexology, auricular acupuncture), for example. The reason that no amount of clinical data can save such claims is that clinical data are necessarily more messy and less voluminous than the data that established the claims as being hopelessly implausible in the first place. Nevertheless, these are being tested clinically, thanks to the NCCAM and other naive EBMers, and guess what the answers eventually, after many false positives and many years and much waste and more, are found to be?

    The second scenario is one in which the claims are not hopelessly implausible, but for various reasons are usually substantially more implausible than most biomedical scientists are used to: injecting or feeding biologically active substances, touted by “CAM”sters, to cure cancer, atherosclerosis, and everything else, sticking needles into people for what ails them, cracking backs (or necks) to lower blood pressure or treat asthma. Again, the claims are being tested clinically, thanks to the NCCAM and others, and guess what the answers are? Yup. But only after the same useless exercise as described above, this time with the added feature of human subjects being tortured and injured.

    Great, huh?

    KA

  8. mark says:

    @ Kimball

    I’ve know you’ve made the point before that you do rely on RCTs and don’t devalue them relative to pre-clinical science when the pretest probability is high, but in this post the distinction is not clear (to me anyway). Billy Joes’ comment does not address the importance of RCTs vs preclinical science when interpreting evidence in that setting.

    I realise you’ve repeatedly posted about what is SBM, but I have tried before to search this website to find the relevant posts and found it hard to find them, which is why I suggested a specific category. You’ve got categories for lots of other things. It seems a shame not to have one for SBM/EBM issues.

    @ Jann Bellamy

    But what about someone who claims that vitamin K or magnesium or whatever is good for bone health based on reasonable preclinical studies, even if it is not supported by clinical studies. The point I was making was that they can say their claims are founded in SBM because SBM says RCTs are overvalued, and cite this post. Of course it would be wrong, but I don’t see anything in this current post that would stop this misuse of the term SBM.

  9. evilrobotxoxo says:

    @Kimball:

    I agree that there are multiple scenarios, and it can get complicated. However, I still think that a major part of the problem is that the current EBM framework presupposes plausibility based on basic science or preliminary clinical observation. I guess one could argue that CAM would claim to fall into the second category, e.g. if this many people claim to have positive results from acupuncture, then we should test it.

    Interestingly, one of the most commonly-discussed topics on this blog is the vaccine-autism link, which is something that was actually quite plausible in the absence of clinical evidence.

  10. evilrobotxoxo says:

    I think Mark makes a good point that SBM is not clearly defined on the web site, unless one searches and reads through a bunch of posts to find it. Maybe there should be a “What is SBM?” link on the front page or something.

  11. Jann Bellamy says:

    @ mark:

    You are right, they could say that but they would be taking the “RCTs are overrated” statement totally out of context. I don’t think one could ever design an epistemology of medicine that couldn’t be twisted, misrepresented, cherry-picked or in some other manner corrupted by the pro-CAM crowd.

  12. mark,

    C’mon. This post says:

    …it probably didn’t occur to Sackett and other EBM pioneers that anyone would consider performing clinical trials of methods that couldn’t pass the muster of scientific plausibility. Their primary concern was to emphasize the insufficiency of basic science evidence in determining the safety and effectiveness of new treatments. In that they were quite correct, but trials of “CAM” have since reminded us that although established knowledge may be an insufficient basis for accepting a treatment claim, it is still a necessary one.

    That is the same thing that my words quoted by Billy Joe say. In the “Yes, Jacqueline” article that I asked you to read, I wrote:

    …everyone here agrees that large RCTs are the best tools for minimizing bias in trials of promising treatments, and that RCTs have repeatedly demonstrated their power to refute treatment claims based solely on physiology, animal studies, small human trials, clinical judgment, or whatever. I made the very point in my talk at TAM7, offering the Cardiac Arrhythmia Suppression Trial and the Women’s Health Initiative as examples…

    To reiterate: The most important contribution of EBM has been to formally emphasize that even a high prior probability is not always sufficient to establish the usefulness of a treatment—parachutes being exceptions.

    How much clearer can I be? There is absolutely nothing in my writing, or in the writing of any of my colleagues here on SBM, to suggest that

    …someone who claims that vitamin K or magnesium or whatever is good for bone health based on reasonable preclinical studies, even if it is not supported by clinical studies…can say their claims are founded in SBM because SBM says RCTs are overvalued, and cite this post.

    The language in this post, quoted above, “would stop this misuse of the term SBM.” Please excuse me if this seems insulting, but do you understand the meaning of “insufficient” and “necessary” in these passages?

  13. mark says:

    @ Kimball

    I’m sorry I’ll drop out of the comments here. I do understand what insufficient and necessary mean.

    I have been a long time reader at SBM, but every time the whole SBM vs EBM comes up, I just don’t get the difference. After your last post, I decided to try and understand as best as I could. I re-read as many of your previous posts as I could find and Sackett’s BMJ editorial. I understand the issues highlighted about pre-test probability and implausibility, which seem to be relevant mainly to complementary medicine. But I really struggle to see any other differences. I won’t comment further.

    I appreciate that you have repeatedly posted on all these issue, but each post is generally read in isolation. To me, the main message of this post is that EBM places too much weight on RCTs and not enough on basic science. As I posted above, I think that someone reading this post in isolation could make that claim and say it is supported by this post because it stresses giving more emphasis to basic science and less to RCTs with little qualification. The sentence you quote is easily missed in a very long post, and readers may not be familiar with your other posts.

    I never suggested you or your colleagues would make such a statement.

    From now on I’ll just skip the whole EBM vs SBM posts.

  14. Dr Benway says:

    Hi mark I just chatted with you in another thread.

    Dr. Atwood’s criticism of much (but not all) EBM is sound, in that prior plausibility has not been given sufficient consideration. This is not merely his opinion but is dictated by Bayes’ theorum.

    Doctors and social scientists have gotten away with ignoring prior odds because the variable becomes moot in a research environment where its value remains similar across comperable studies. In such an environment frequentest statistics are good enough.

  15. Dr Benway says:

    Oh, I forgot to say that “too much emphasis on RTC” verses “not enough emphasis on RTC” is a red herring. You have to ask, why do we value RTCs so much? Politics? Wat?

    Really, RTCs are nothing more than an attempt to control for confounding variables. *shrugs*

    Sadly, amazingly, incredibly, all evidence is important, whether of the RTC form or not. All relevant evidence relevant to a proposed hypothesis must be weighed if we want the highest level of confidence in our conclusion about that hypothesis.

  16. Dr Benway says:

    I still think that a major part of the problem is that the current EBM framework presupposes plausibility based on basic science or preliminary clinical observation. I guess one could argue that CAM would claim to fall into the second category, e.g. if this many people claim to have positive results from acupuncture, then we should test it.

    And thus the CAMsters would reveal their tunnel vision.

    Consider these two hypothesis:

    1. Homeopathy is true.
    2. Homeopathy is not true.

    Let’s assign a probability to the first hypothesis and call it x. The second hypothesis must necessarily then have a probability of 1-x. Note that increasing the probability of either hypothesis decreases the probability of the other.

    Now, everything currently understood about physics and chemistry stands in support of the second hypothesis and in contradiction to the first. So if we increase x on the basis of a few positive patient reports or a few surprising RTCs, we must be mindful of the fact that we are at the same time assigning a lower level of confidence to all that has been established within the basic sciences to date.

    Only a very egocentric individual would feel entitled to pass judgment on the complex, peer reviewed literature published in physics and chemistry, simply on the basis of a few patients testimonies or the word of a family member who happens to sell Herbalife.

  17. dental dam says:

    I DO agree that there are many many scenarios, and it can get complicated.

    But, I still think that the key problem is that the current EBM framework presupposes plausibility based on basic science or preliminary clinical observation. I guess one could argue that CAM would claim to fall into the second category, for example. if this many people claim to have positive results from acupuncture, then we should test it.

  18. JMB says:

    I think that Dr Atwood has provided an excellent description of the errors in the formalism of EBM. Those errors in the formalism prevent low probabilities from being effectively used in a decision analysis to indicate the lack of benefit of CAM treatments. What good is a formalism that doesn’t allow refutation of a hypothesis with both low a priori and low a posteriori probability?

    I would suggest another way to explain the difference between EBM and SBM; the recognition of the placebo effect. The formalism of EBM could be more complete (and closer to SBM) if it would include the principle that a scientifically acceptable mechanism of action must be determined for a treatment to not be labeled a placebo. By denigrating basic clinical science, EBM has lost one of the primary classifiers of placebo effect. Perhaps that is why EBM cannot declare homeopathy (and several other equally implausible CAM therapies) placebo effect.

    As suggested in a comment by Always Curious, when a strong unexpected effect is noted in a clinical trial, it can become an incentive for basic clinical science research to explain a mechanism of action. There is no reason under EBM to evaluate an observed effect for mechanism of action.

  19. BillyJoe says:

    mark,

    Maybe this will read better for you:

    Prior Probability is a NECESSARY, but generally not SUFFICIENT, basis for incorporating a treatment into general use. For that you also need Clinical Trials.

    That puts the emphasis on both prior probability and clinical trials. It’s not a fight of one against the other.

  20. Dr Benway says:

    A corollary: “RTCs to test hypotheses of very low prior probability will send our country on expensive wild goose chases, as any positive results will be wrong.”

  21. Dr Benway says:

    Sorry I’m spamming with all these posts, but I’m searching for the best way to explain the importance of prior odds.

    Analogy:

    The San Diego PD arrive at a home after someone has been shot dead. They see a tall figure running from the back of the house. They give chase but lose him. Later while patrolling the neighborhood they see a guy who looks like the runner. They arrest him and send his DNA samples off to the lab. If the test comes back positive, the cops have their killer, correct?

    Ok, same story as above. However, this time the cops don’t chase anyone. They simply pull a random name of someone living in China from a large database of everyone living on Earth. They get a tech in China to draw the guy’s blood and send it off to the lab.

    Now if the Chinese guy’s DNA matches samples from the crime scene, will we lock him up for murder?

    You look at the package insert for the DNA test and read an estimated false positive rate of about a million to one.

    Wait, what? False positives can’t be the same for story one and story two. Story two *must* have a false positive rate of 100% or 1:1.

  22. JMB says:

    …someone who claims that vitamin K or magnesium or whatever is good for bone health based on reasonable preclinical studies, even if it is not supported by clinical studies…can say their claims are founded in SBM because SBM says RCTs are overvalued, and cite this post.

    Consider in the decision to treat that EBM values basic science .01 and RCTs .99. In the decision to treat, SBM values basic science .1 and RCTs .9. I think those formulae are pretty basic simplifications of the process, but serve to get the point across. In the decision to treat, SBM does not test tube (or petri dish) results to being equal in importance to RCTs. Basic science findings do not justify treatment when there are negative results for benefits in clinical trials. In language descriptions we rely on ambiguous phrases (ambiguous in a mathematical sense) to convey the relative value of factors. Of course, Dr Atwood is really defining boundary (necessary) conditions for the decision rather than fuzzy estimates for the factors.

    In the Bayes paradigm, the experimental design provides a likelihood ratio. That likelihood ratio multiplies the prior probability of the hypothesis to arrive at the posterior probability after the experimental result. There is limit to the power of an experimental design, a frequentist concept that I think relates to the limits on the likelihood ratio. When the a priori probability is less than .0001, it is hard to imagine any experiment having enough power to raise that probability to .1 in the posterior result. However, when the prior probability is .7, it is easy to imagine an experiment with enough power to result in a lower posterior probability of .1.

  23. JMB says:

    “In the decision to treat, SBM does not test tube (or petri dish) results to being equal in importance to RCTs.”

    Sorry that should read, “In the decision to treat, SBM does not raise the value of test tube (or petri dish) results to being equal in importance to RCTs.”

    Instead of saying negative results for benefits I should have said null results. Negative implies that the experimental arm fared worse than the control arm of a clinical trial.

  24. JMB says:

    I am as harshly critical of the hierarchy of evidence as anyone. I see this as something that will self-correct over time, and I see people within EBM working both formally and informally to replace the rigid hierarchy with something that places each research study in context.

    I would agree that it will self-correct over time, but remember time can be a very precious resource for patients. Many can die waiting for self-correction. For a practicing doctor, it is unethical to wait for correction, it is better to slam the correction into place. It is also a waste of resources to wait for self correction. Personally, I think EBM has been so badly tainted by woo, that it is time to replace the moniker and start over again as SBM, incorporating the concepts put forth by the editors of this site.

  25. Dr Benway says:

    When the a priori probability is less than .0001, it is hard to imagine any experiment having enough power to raise that probability to .1 in the posterior result. However, when the prior probability is .7, it is easy to imagine an experiment with enough power to result in a lower posterior probability of .1.

    That’s good, Imma translate it into grandma-speak:

    If the prior odds of some therapy being effective are less than 1/10,000, any study with a positive outcome will almost certainly be a false positive. When the prior odds are closer to 7/10, we’ll still get false positives, but mostly true positives.

    I think the language below will confuse the n00bs:

    Consider in the decision to treat that EBM values basic science .01 and RCTs .99. In the decision to treat, SBM values basic science .1 and RCTs .9.

    This language implies that a political struggle exists between two parties. Big mistake! A mathematical “value” is not a political “value.”

    Also, I think it confuses people when we talk about the relative emphasis or “importance” of RTCs. By “importance” we mean statistical power. However the lay public imagine we’re talking politics.

    In mathematics you’re not an assh0le when you say, “my answer is correct and yours is wrong.” In politics, you are.

    Anyone who understands Bayes’ theorem can see that the SBM guys are right and their opponents are simply wrong.

    So why the arguing? Because two decades of alt med propaganda have normalized woolly headed thinking among the faculty of our medical schools. That fact is so incredible and so difficult to accept that, well, it will take time for people to get their heads around it.

  26. JMB says:

    @Dr Benway

    You’re right. I should have called it a decision factor, not a value. Calling it a value confuses it with personal or public values. EBM assigns a factor of .01 to basic science results.

    I think EBM tries to oversimplify (as I have done), and assign a factor in the decision a number based on methodology. In the old medical decision making field, the assigned numbers would vary based on the strength of evidence from different methodologies, as well as individual patient factors. That is why it is difficult to simplify the SBM approach to the same degree as EBM. It is not only the Bayes vs frequentist approach, it is the complexity of the decision process in SBM versus the simplicity in EBM.

  27. enosh askenasy says:

    I think the SBM vs. EBM dispute is mostly a matter of resolution.
    EBM is mode of thinking suited for hordes while SBM strives to direct individual clinical decisions.

    Mr. Simon (which I understand is a statistician) see’s clear facts as the gold standard of research. The multiple variables that make up an individual patient are for him “white noise” that needs to be minimized (most effectively by PRT). And the gathering of facts to form a scientific theory ,greater then its sum of parts, introduces an implicit uncertainty which lowers the statistical certainty (giving basic science its dismal location in evidence hierarchies).
    Physicians like Dr. Novella on the other hand, are focused on individual clinical decision making. They are interested in a mode of thinking that will help in real-life diagnosis and treatment of patients and research that will advance their decision making.

    No one (rational) is disputing the importance of EBM, best exemplified by the thalidomide example. A drug given arbitrarily to Multiple Myeloma patients without any scientific rational and found to be effective. In fact most of our medications were discovered more by luck than reason.

    Importantly though, there was no science to indicate the contrary to the use of thalidomide in MM. Likewise, no one would try to actually test the parachute hypothesis by jumping of planes without them, because we have a basic scientific understanding of gravity (I think researching homeopathy is silly).

    SBM basically sais that in the context of an individual patient EBM must be taken into account but moderated by our knowledge of physiology and science, this moderation is vital since patient uniqueness erased by PRT is crucial in a horde of one.

  28. anoopbal says:

    Just like how so many treatments was approved or rejected by looking at meta-analysis and systematic reviews though an EBM approach, why can’t it happen to homeopathy and acupuncture too? Is it the failure of EBM or it the insistence of these CAM groups to intentionally distort the evidence.

    And when you talk about plausibility and is all about the treatment not violating the established science, the problem is that we don’t know if the explanation used is wrong -the observations might be true.

  29. Dr Benway says:

    And when you talk about plausibility and is all about the treatment not violating the established science, the problem is that we don’t know if the explanation used is wrong -the observations might be true.

    Your language is confusing me a little. Can I replace “explanation” with “hypothesis” and illustrate with an example?

    Hypothesis: “homeopathy works”
    Observation: an RTC showing a positive effect for homeopathy against placebo with a p = .05.

    Yes, the observation is correct –meaning no fraud– in my example. But we *do* know the hypothesis is wrong, because of physics.

  30. DevoutCatalyst says:

    “Because two decades of alt med propaganda have normalized woolly headed thinking among the faculty of our medical schools.”

    It’s a woolly mammoth.

  31. Dr Benway says:

    I think the SBM vs. EBM dispute is mostly a matter of resolution.
    EBM is mode of thinking suited for hordes while SBM strives to direct individual clinical decisions.

    Uh, no. Prior probability. That’s it. Don’t muddy the waters.

  32. anoopbal says:

    [i]Your language is confusing me a little. Can I replace “explanation” with “hypothesis” and illustrate with an example?
    Hypothesis: “homeopathy works”
    Observation: an RTC showing a positive effect for homeopathy against placebo with a p = .05.
    Yes, the observation is correct –meaning no fraud– in my example. But we *do* know the hypothesis is wrong, because of physics.[/i]

    No.

    Explanation means the science used to come to the “hypothesis”. For example, acupuncture might be working with the release of endorphins which is plausible, but the CAM experts would say it’s working via meridians and energy flow which is not plausible. So the date might be true, but the causal mechanisms might be wrong here.

  33. Dr Benway says:

    . EBM assigns a factor of .01 to basic science results.

    For the sake of those who haven’t read closely, JMB doesn’t mean literally “.01.” That’s just a made up number. The point is, EBM’s pyramid of evidential merit had “basic science” down low.

    There’s a bigger problem. “Basic science” as a term is a lie. It has nothing to do with Nature. It’s merely a social construct humans find convenient.

    Imagine a parallel universe in which the NEJM publishes about three physics or chemistry papers each issue. Now do your systematic review of homeopathy.

  34. Dr Benway says:

    So the date might be true, but the causal mechanisms might be wrong here.

    OK, but “causal mechanism” is where the equivocators hang out. You must anticipate them else you’ll have Dianetics rather than psychiatry at your hospitals.

    Consider these two syntactically parallel sentences:

    1. We don’t know how homeopathy works; we just observe that it does.

    2. We don’t know how Prozac works; we just observe that it does.

    There is a gross epistemic difference between 1 and 2. If it isn’t immediately obvious to you, then I have a bridge to total freedom you might be interested in purchasing.

    When you say, “x works” you must define what you mean by “x” in concrete terms your grandmother can understand. If you can’t do that, you are simply saying “something works.” That and a cup of coffee will get you a cup of coffee.

  35. Gabor Hrasko says:

    “For example, acupuncture might be working with the release of endorphins which is plausible, but the CAM experts would say it’s working via meridians and energy flow which is not plausible. So the date might be true, but the causal mechanisms might be wrong here.”

    But endorfin is released regardless where you prickle the patient. Plausibility question should be applied to the claim of acupuncture, i.e. that there are specific points that are active. Prickling the patient does not equal to acupuncture.

  36. Dr Benway says:

    Prickling the patient does not equal to acupuncture.

    Exactly.

    Whenever you hear, “x works,” be sure to find out precisely what is meant by “x.” Sometimes it means, “mystery sandwich.”

    DO NOT WANT MYSTERY SANDWICH!!!! BEWARES!!!!!

  37. daedalus2u says:

    The goal of medicine is not to formulate correct hypotheses, but to administer treatments to patients so that the health problems that patients have get “better”.

    The patient getting “better” has several components, including the patient feeling better and also the patient experiencing an improved state of health via objective outside criteria. If all that was necessary was the patient’s subjective feelings, then administering a combination of cocaine, heroin and amphetamine would be the “best” treatment because no matter what the patient had it would make them feel really good.

    Treatments that only influence subjective symptoms must be looked at extremely carefully to ensure that they are not simply acting the way that a combination of cocaine, heroin and amphetamine would act. One way of doing that is with clinical trials that look at objective endpoints. Another way is to understand the underlying physiology and how the treatment affects the underlying physiology.

    A clinical trial that only looks at subjective endpoints (the patient feels better) is analogous to a trial of cocaine, heroin and amphetamine. If such a trial was made short enough, all patients would do very well. Fortunately we know some of the physiology behind administering a combination of cocaine, heroin and amphetamine and SBM would not consider such a trial in the first place. EBM can only look at what trials have been published, so if a supplier “games” the medical literature, bad treatments can have positive published clinical trials.

    That is what CAM is doing now, “gaming” the medical literature and the clinical trial procedures to get their trials with only subjective endpoints considered.

  38. JMB says:

    @Dr Benway

    You’re right again. 0.01 is just a fuzzy logic value (value in the sense of fuzzy logic theory) estimated from some of the published Cochrane articles that seem to lack input from other scientific disciplines (or for that matter, diverse clinical scientists). That’s a statement of my opinion. The formula was intended to demonstrate that SBM can increase the weight of basic science but avoid the trap of recommending treatments that consistently fail RCTs in spite of the body of evidence from more basic science. Although the formalization of EBM does not include such a formula, the relative weights can be inferred by the hierarchy of evidence. Fuzzy logic values can be inferred from the accepted papers published on the Cochrane sites (although those fuzzy logic values may not be what Sackett intended). The result of such a fuzzy logic assessment is not that we can quantitatively compare SBM and EBM, but that we can increase the importance of basic science evidence without completely devaluing consistent evidence from RCTs.

    However I’m sure I bored most reader’s with the more rigorous description (even more than were bored with the less rigorous description).

    **********************

    # enosh askenasy

    I think the SBM vs. EBM dispute is mostly a matter of resolution.
    EBM is mode of thinking suited for hordes while SBM strives to direct individual clinical decisions.

    I think the editors on this site are also saying that the Bayesian approach should also be applied to the assessment of published research, and to decisions of allocation of public resources. It will be interesting if Ioannidis association with the Cochrane Collaboration will have such an impact.

    My personal bias is that Sackett took some elements of medical decision making theory (that part evaluating how medical experts assessed published research) and dropped the rest (how research results can be translated into medical decisions). The Bayes approach had a big impact on the field of medical decision making. SBM would restore many of the Bayes implications of how medical research should be assessed, how resources should be allocated for research (and what constitutes clinical equipoise), and how we make decisions in medical practice based on assessment of research.

    I also think that the trend toward incorporation of individual genomes into medical decisions will force some changes into the traditional hierarchy of evidence. RCTs will be necessary to prove a causal link. But individual decisions will require a Bayes approach because of the amount of information will will have to predict an individual’s physiologic response to intervention. Our risk vs benefits calculations will in many cases be based on individual factors, rather than matching limited individual characteristics to the selection criteria in randomized clinical trials.

    *****************************

    This is somewhat off subject, but is related to these comments. The recent USPSTF used fuzzy logic values (I am not saying their reasoning was fuzzy in a derogatory sense, but shared characteristics with fuzzy set theory). Although the calculated risk vs benefit estimate was 1 in 1904 lives saved in the age group 40 to 50 for screening mammography, they did not recommend screening because of poorly quantified risks of overdiagnosis and overtreatment. They were also recommending a fuzzy logic based decision (again a mathematical model, not a derogatory term) that women (aged 40 to 50) consult with their doctors for individual determination of risk factors that would play into the decision. My primary objection (not my only objection) was that there was enough data from the RCTs that the frequentist estimates of the parameters was sufficient to indicate that those fuzzy estimates had already been quantified, and included in the estimate of risk vs benefit of 1 in 1904 lives saved. The frequentist approach can give us reliable estimates that we can incorporate in a Bayes analysis. When reliable probabilities are available, we should use those probabilities rather than fuzzy logic estimates.

  39. enosh askenasy says:

    Dr. Benway,
    Didn’t mean to muddy the water, I was trying to make an observation, obviously badly phrased.
    What I basically meant was that SBM is in tune with the way physicians think. Prior Probability is the tool of our trade when diagnosing, it stands to reason that we will see it as a requirement for research fund appropriation. To a statistician’s way of thinking this is a foreign concept. You cannot numerically quantify prior probability, you cant build an algorithm around it (in my limited understanding). Its basically akin to “common sense”.
    In the era before EBM the physician was at the center of medicine- research and clinical decisions. EBM’s importance is in its standardization of medicine trying to limit human error, In the process an unintentional “blind spot” as you called it developed.
    Simply put, if physicians don’t know best than the nature of medicine is up for debate. If the field of battle is randomized trials we can study everything and anything (Alien crop circles for pancreatic cancer). If medicine is a statistical entity then hard facts are a priori, and it is in the nature of statistics to negate prior assumptions in the favor of concrete results.

    JMB
    I think the discussion about personalized medicine in the era of genetics is fascinating. But as you said, it is outside the scope of this discussion. Perhaps someone at SBM will “pick up the glove” and write about it.

  40. rork says:

    The trouble not being addressed sufficiently (for me) by advocates here is how to obtain the prior. Call it plausibility or what-you-will since the words don’t matter, people will not agree on it. No mechanism to obtain it is proposed. No procedure to elicit it is proposed.

    Trying to summarize what the bayes factor is may be what EBM is de facto doing. You thereby avoid fighting about the prior, which is supplied by the individual, and will vary. You can conclude the evidence (bayes factor) is small, and may end up writing it needs more study, but I read some of those sentences as “if you want anyone to believe this crap you’ll have to come up with more evidence, cause right now it sucks”, and not as statements that “the public should fund more studies of this”. I agree the authors of such things should permit themselves to be a bit more outspoken.

    I agree basic science is part of the evidence, but it is very hard to quantify how it affects the bayes factor obtained from, say, clinical trial data. We are very poor at making those calculations. I can say something about why, but it starts to be a long story having to do with how many possible models for mechanisms to explain a thing there are, and that specifying that prior would take infinite time, which would slow down science a little.

    enosh: “To a statistician’s way of thinking this is a foreign concept. You cannot numerically quantify prior probability”. Perhaps I don’t follow, but for a bayesian, that’s exactly what you do when it’s you making a decision. (Imagine we bet on the outcome of tossing a coin repeatedly or even just once – I guarantee I will be using an exact prior, and it’ll happen to have a beta distribution.) If on the other hand you are trying to convince the public, you often say something about how different the posteriors would be depending on differences in the priors, since you admit people will not agree on it. I wish some people here would admit people don’t agree about priors, and admit that this is a problem. Instead, what you will get back is anecdotal examples, as when the prior is “essentially zero” for the writer at least – but it’s not empirically determined to be true for any group of people, and no mechanism to determine it is proposed.

  41. I thought that if you have enough research, that being able to calculate an original prior isn’t necessary.

    Say I arbitrarily start with the assumption that the prior probability of homeopathy being able to cure pancreatic cancer is 50%. This is before I have looked at any research at all. Then I take that 50% and multiply it by the probability of the existence of water memory, and multiply again by the probability that water memory can be transferred to dried sugar granules, and again by the probability of a molecule of active ingredient being present in the remedy, then by the probability of magic being a real force in medicine, then finally by the probability that your RCT of homeopathy and pancreatic cancer is showing a positive result.

    Or say I start with the assumption that the prior probability of homeopathy being able to treat anything at all is 50%, then multiply by the probability of all existing RCTs of homeopathy showing a positive effect.

    (My appalling lack of basic statisical understanding is showing. I am sure someone will kindly figure out what I’m trying to get at and restate it as something that makes sense. Which the above does not, I realize.)

    But anyway, I thought that you didn’t need to know the original prior; 50% is good enough because with homeopathy by the time you’ve reviewed the relevant research you’re going to end up with something fantastically miniscule anyway. And for something like “probability of water memory being transferrable to dry sugar granules,” since there is no supporting evidence at all you can arbitrarily (and exceedingly generously) set the probability at 0.001%. You will still end up with a fantastically small probability that any apparent positive result of an RCT reflects activity of homeopathic remedies.

    (… is there any sense to extract here? Or am I just missing the point?)

  42. enosh askenasy says:

    rork
    I stand corrected.
    In many instances relating to CAM a bayesian statistician will find it takes an infinite time to calculate the prior probability derived from basic science, a practicing md such as Dr. Benway will do it in 3 seconds without using a calculator.

  43. Dr Benway says:

    You cannot numerically quantify prior probability, you cant build an algorithm around it (in my limited understanding). Its basically akin to “common sense”.

    Everyone accepts that in medical decision making, gray areas exist where reasonable people may disagree. There’s no point in trying to make that problem go away.

    But the fact that gray areas exist should not distract us from a more pressing problem: medical batsh1t.

    With respect to batsh1t, you certainly CAN numerically quantify prior probability. There are several ways to do this. One I illustrated someplace around here recently: look at the published evidence that contradicts the hypothesis in question and calculate the statistical power required to overturn all that.

  44. JMB says:

    @rork

    The trouble not being addressed sufficiently (for me) by advocates here is how to obtain the prior.

    Sometimes I think EBM is a design by epidemiologists frustrated in trying to work with clinical scientists. For reasonable priors for medical hypotheses being researched, clinical scientists can be polled. One requirement is that the clinical scientist must be well versed in the field. A second requirement is that several qualified clinical scientists must be polled to obtain a sort of consensus. Statisticians may not like the imprecision of the estimates, but even with the imprecision, the quality of the resulting decisions can improve. It only takes a ball park estimate of .0001 with confidence intervals of .01 to .000001 to show that any positive benefit observed from administering a homeopathic remedy must be a placebo effect. In spite of the imprecision, that is still a better conclusion than reached by EBM.

  45. Dr Benway says:

    JMB, I’m not sure I like the polling argument. I think we can do better than that.

    Often we can use a little logic to reason out the implications of some hypothesis, and often we do have evidence pro or con those implications.

    Intersubjective agreement should be a last resort.

  46. BillyJoe says:

    ‘I thought that if you have enough research, that being able to calculate an original prior isn’t necessary. ”

    The problem is that you want to actually avoid doing that research when the prior is close to zero. With limited time, money, and manpower, you have to make choices. That sort of makes the priors essential.

  47. JMB says:

    @Dr Benway

    we can use a little logic to reason out the implications of some hypothesis

    I would agree with that. I was just trying to anticipate the counter argument that we would be picking priors based on our bias of what the expected outcomes should be. Based on my experience in computer aided diagnosis, if you qualify experts by measuring their success in similar decisions, you can arrive at a close approximation of the prior probabilities by intersubjective agreement that would be measured if you had the resources to conduct the formal observation. The testing of the experts is a slight modification of the intersubjective agreement process that improves its outcome. A lot of people can be in a position that would qualify them as experts, but measured performance on certain types of decisions would be a better qualification of expert status. The qualification process can counter the argument of bias. If Dr Atwood can successfully predict the outcomes of controlled clinical trials of homeopathy as hovering around the result of placebo effect, then it is hard to dismiss Dr Atwood’s estimated priors as bias.

    Priors can be estimated by a few methods. They don’t have to be that exact to be useful. Low priors that exclude the possibility that we are observing anything other than a placebo effect don’t have to be exact. They can definitely fall into the grey area of precision. The priors (and posteriors) for which we need more exact estimates are those in which we are proposing intervention in the general public, for example the screening interventions. Small errors in precision can end up affecting large numbers of people (1% of 22 million is 220,000).

    It is interesting that the government tried the qualification process and intersubjective agreement in predicting terrorist attacks. They used a sophisticated method based in game theory of identifying which analysts would have the best success in predicting attacks. Unfortunately, it got smeared in the public press as a ruthless gambling office pool in which people won money when they correctly predicted a terrorist attack. The public failed to understand how sophisticated a system it was for qualifying the analysts as experts, and thereby improving the precision of estimated priors (it was an intersubjective scheme). I guess you could say that the success of bookies is a measure of success of intersubjective agreement.

  48. Dr Benway says:

    Yes, I understand signal averaging as a way to reduce noise. It’s a good idea. But a last resort. The vitaministas have become very wealthy and politically connected. They have figured out how to stack panels of experts in their favor.

    Anyway, why do all that when we can simply say, “If homeopathy is true, the Standard Model is wrong. The evidence in support of the Standard Model is 19 billion papers with p<.0001. Therefore, any RTC study of homeopathy will require x subjects to achieve sufficient statistical power to overturn all of that, with x being a number so huge your head would explode if you saw it."

  49. JMB says:

    It’s more than signal averaging to reduce noise, it is a way of identifying which of those we call experts can give us fairly accurate estimates of prior probabilities. So it doesn’t average all estimates, just estimates of those with a proven track record. Furthermore, since the measure is based on prospective (or blinded to results) estimates of prior probability, then the selection process counters the criticism of bias in the panel (and there are statistical measures of bias that can be derived from the selection process). So Kimball Atwood should apply, but Dana Ullman needn’t bother. But if both were candidates, we could be comfortable that the selection process would reliably select the true expert. It is also a way for NCCAM in the NIH to dig itself out from the burial by woo.

  50. JMB says:

    Wouldn’t it be interesting if the editors of SBM were given the opportunity to predict the results of government funded experiments in CAM, and by weight of success of predictions, could eventually overrule the current panel used by NCCAM to grant funding for studies. That would certainly be in the interest of taxpayers. Or you could set up a system like the CIA, and allow anyone to bet real money on the outcome of proposed studies. Then the greater your winnings, the more your input would be weighted for granting funding of research. I’d rather bet on NCCAM results than the stock market (who wouldn’t?).

  51. Dr Benway says:

    JBM,

    Prior to becoming director of NCCAM, Dr. Josephine Briggs had a distinguished career as a genuine scientist with published work in nephrology.

    My point is, the pro-woo crowd is very good at finding compliant experts. They’re just a lot smarter than we are politically (srsly, how many here give money to politicians regularly?) And, unlike bigPharma, they’re not under the watchful eye of a broader community of academic physicians.

    The “panel of experts” idea is no guarantee against the rise of Dr. Lysenko. So, if we have a more objective means of separating sense from nonsense, we should use that first.

  52. geo says:

    Sorry for not wading through the prior comments, but has anyone mentioned the danger of allowing prejudices and instinctive beliefs to affect our understanding as to what is plausible?

    Evidence based medicine certainly has its flaws, but that forces us to acknowledge just how limited our current understanding of many medical matters still are. I worry that the presumption that we are capable of determining what is plausible and what is not will lead to a legitimisation of those instinctive beliefs to be found amongst amongst the medical community: those we all have, those especially common amongst those drawn to work in medicine, and those inculcated by the experience of medical practice.

    If there is evidence showing that homeopathy can work as well as other, more plausible treatments in RCTs, then that’s good reason to look again at the plausibility of these more conventional treatments imo. RCT’s themselves have limitations, especially when it comes to subjective ailments, but this is a problem for plausible and implausible treatments alike, and some of the talk about science-based-medicine sounds worryingly like trying to put a finger of the scale because of prior expectations. Surely a poor way to do science.

    I just stumbled upon this quote yesterday: “Science is the organized skepticism in the reliability of expert opinion.” – Richard Feynman.”

  53. Mojo says:

    If there is evidence showing that homeopathy can work as well as other, more plausible treatments in RCTs…

    And if there isn’t?

  54. geo says:

    “And if there isn’t?”

    Then plausibility doesn’t need to come in to it. Evidence based medicine did fine without it.

  55. Chris says:

    Um, geo, how is it plausible for a chemical interaction (like curing influenza) with something diluted to one part to more parts than there are atoms in the universe?

    Homeopathic Oscillococcinum 200C is one part duck bit to 10400 parts of solvent. There is only an estimated 1080 atoms in the known universe. So please explain how in this universe Oscillococcinum would have a plausibility of actually affecting a person’s biological reactions to influenza.

  56. geo says:

    I’m not saying anything about the plausibility of homeopathy, which is almost certainly rubbish. I don’t know why you think I am.

  57. Chris says:

    Aren’t these your words:

    If there is evidence showing that homeopathy can work as well as other, more plausible treatments in RCTs…

  58. geo says:

    Yes.

    More completely: “If there is evidence showing that homeopathy can work as well as other, more plausible treatments in RCTs, then that’s good reason to look again at the plausibility of these more conventional treatments imo.”

    So if similarly designed RCTs to those used as evidence for the efficacy of a conventional, plausible treatment could also be used to show homeopathy had a similarly beneficial effect, then we should look closely at the study design and examine whether the plausible treatment is really having any positive affect. If we were to have higher standards for the evidence needed in support of homeopathic interventions then that could allow plausible sounding treatments to be used despite not being any more effective than homeopathy.

    By empahsising plausibility, rather than just evidence, science based medicine could make medicine less scientific and more easily influenced by the distortions of thought of its practitioners and researchers. (Also, as useless medicines go, at least homeopathy doesn’t have side-effects).

  59. Chris says:

    Okay, tell us what conventional treatments need to be checked for plausibility when compared to homeopathy. Because by bringing up the false dichotomy of “If there is evidence showing that homeopathy can work as well as other, more plausible treatments in RCTs” you are really confusing the issue.

  60. daedalus2u says:

    geo, did you read the post? It discusses what is meant by “prior plausibility”. It does not mean what you think it means.

    The problem with doing clinical trials on random things with no rational basis behind them (such as trials on homeopathy) is that RTCs are done on human beings, and human beings need to be treated respectfully and ethically and according to the Declaration of Helsinki. Otherwise it is a crime against humanity.

    For a RTC to be done on humans, there must be “clinical equipoise”, that is the two legs of the trial must be clinically equivalent. One cannot be obviously clinically inferior. This is why a vaccinated/unvaccinated trial cannot be done. It is unethical to not vaccinate children against vaccine preventable diseases because those diseases are known to be much worse than the vaccine.

    By what basis is the magic water produced by homeopathic principles thought to be clinically equivalent to any treatment with actual medicine?

  61. geo says:

    @ chris: I was only speaking about abstract possibilities. Y’know – for philosophical fun. I was reading this a couple of days ago though: http://jme.bmj.com/content/early/2010/10/20/jme.2010.039313.abstract

    @ daedalus2u:

    So is the plausibility argument only used for deciding research funding then? I thought I’d read other articles here were it was being promoted as a way of evaluating the evidence from research? Maybe I am mistaken though – I had thought the article above said that the Cochrane review found positive RCTs for asthma and homeopathy.

    I actually think that political considerations and the desires of patients should be given significant influence over decisions about research priorities, but that is a rather different discussion.

  62. Chris says:

    I am an engineer, I hate philosophy. Try something based on reality.

  63. BillyJoe says:

    “I actually think that political considerations and the desires of patients should be given significant influence over decisions about research priorities…”

    Yes, because politicians and patients know so much more about medicine than actual doctors do, so it’s only reasonable that they make the decisions about what gets research dollars.

  64. Dr Benway says:

    Prior plausibility is based upon evidence. In homeopathy’s case, that evidence includes everything we know about physics, which is a lot.

  65. anoopbal says:

    I think the problem here is the Science based medicine is not well defined here. I think as someone suggested have a post which talks about what is SBM and what are the common criticisms and link on the About page.

    Like,
    What is SBM and EBM?
    Why we need SBM?
    What are the common criticisms of SBM?
    What are the limitations of SBM?
    Examples?
    Have a flow chart explaining the steps to see which treatment is worth by the SBM definition.

    I think these longs post without too many subheading makes it harder to figure out things,especially for newbies. I think all the good points get hidden in these longs posts.

  66. daedalus2u says:

    Geo, you misunderstand again. SBM uses all available information all the time. SBM does not compartmentalize evidence and use it for only one thing and then discard it. SBM uses and continues to use all available information at all times and for all purposes. Data that doesn’t fit together raises questions that need to be answered until all the data does fit together. SBM resolves that by getting more data, doing more experiments, doing more analysis.

    When SBM looks at the data on homeopathy, everything fits with the hypothesis that homeopathy is a placebo. Basic research and clinical trials point to the same understanding that homeopathy is a placebo and nothing more than a placebo. Clinical trials with selective publication are unreliable for determining if something is a placebo or not.

    If homeopathy did “work”, then we live in an acausal universe where magic works. I understand that many people want magic to “work”, so they can wish for things and expect them to happen. That is the basis for The Secret, the power of prayer, and the motivation behind virtually all CAM. If we actually did live in an acausal universe where magic “worked”, then there would be hints of that in basic research. There are no such hints. Anecdotes of magic working are best explained by people wanting to believe that their particular magic works and so they misinterpret what happens as being caused by what ever magic they were trying to invoke.

    The requirement for prior plausibility before clinical trials can be done on humans is not about not wasting research funding, it is about treating the humans in that clinical trial ethically. If politicians can waste taxpayers money on useless wars and tax cuts for the extremely wealthy, they can waste it on useless research projects too.

  67. geo says:

    Hi daedalus2u:

    It’s possible we’re misunderstanding each other, but I’m still not really clear as to how SBM improves upon EBM in it’s analysis of the available evidence. I understand why plausibility should be a guiding factor for deciding research funding (although this is another complicated topic, and I think that it is the desires of society and patients which should determine the aims of publicly funded research rather than the interests of scientists).

    I don’t think there’s any realistic chance that homeopathy is anything other than placebo – but having a placebo which its proponents think is effective and want evidence that this is the case could be a more useful control than an unloved sugar pill. That respectable journals have published papers on the positive affects of alternative, treatments with seemingly implausible mechanism (often for conditions of which subjective assessments are required to measure impact of treatment) should mean that we are equally sceptical of plausible sounding treatments whose efficacy is supported by similar papers.

    I’d prefer to talk in abstract about these philisophical approaches to medicine, but for the engineers amongst you: CBT is widely recommended for the treatment of CFS. In RCT’s patients who have received CBT tend to fill in questionnaires in a way which reports lower levels of disability (I’m ignoring all the problems with diagnosing a condition like CFS – the trouble with real world cases is that they’re always too complicated to talk about concisely). This paper followed up one of these trials showing CBT to be marginally effective: http://www.ncbi.nlm.nih.gov/pubmed/20047707 It used actometers to measure activity levels amongst patients, and found that there wasn’t a significant difference in activity levels between CBT and control group (although this was not mentioned in their initial paper). The writers of the piece, CBT proponents, concluded that CBT did effectively reduce fatigue levels (because of the questionnaires) but not through the mechanism of increasing activity. To me, it seems entirely possible that CBT was able to alter the way people answered a questionnaire, but had no real impact upon levels of ‘fatigue’ or disability.

    I don’t really want to get in to all the complexities that surround CFS research, but I think that there is a danger that the fact CBT seems so very plausible to many within the medical community means that it is being over-promoted relative to the evidence we have in support of it’s efficacy. I think that there’s a real danger that ‘plausibility’ leaves room for prejudices and prior assumptions.

  68. JMB says:

    @Dr Benway

    So, if we have a more objective means of separating sense from nonsense, we should use that first.

    There is a difference between experts defined by credentials, and experts defined by results of tests of performance. By testing people who have research credentials for their ability to correctly predict the outcomes of experiments, promoting those who pass the initial test, and continued monitoring of their performance if they are promoted to the panel of experts, the process can be made more objective. It is an emphasis on outcomes as a test of whether an “expert” should continue to be paid for their decisions.

    @geo

    but I’m still not really clear as to how SBM improves upon EBM in it’s analysis of the available evidence.

    It is partly an optimization problem. If we value plausibility to highly, we become entrenched in current concepts and don’t move forward. If we value plausibility too lowly, we waste taxpayer dollars on bogus research, and allow too many preliminary results to become standard of care. The optimum classifier (plausible or implausible) will allow basic research into less plausible studies based on the availability of resources (taxpayers dollars) but erect barriers to the idea that each new RCT replaces the results of every previous RCT. It is not that hard to find a more optimum approach to the problem than currently suggested by EBM. The other side of the coin is that there are innumerable plausible hypothesis that were quite promising in lower level research that never panned out in RCTs. The end result is that the plausible hypothesis is discredtied as a viable treatment.

    There are dangers in trying to submit placebo effect to RCTs. There is tremendous variability in observed effects, and the behavior of the healthcare provider has a big impact on the result, as well as the beliefs of individual patients. That will likely be a real money pit, as RCT after RCT demonstrates conflicting results. While certain placebo effects will record more powerful results (such as midline sternotomy and ligation of the internal mammary artery to alleviate debilitating chest pain), that does not mean the result of the RCT should be used in the decision to use which placebo. Ethics, economics, and the ability of the healthcare providers to keep a straight face, are better deciding factors.

  69. daedalus2u says:

    I am happy to get into a discussion of CFS and CBT. I know a great deal about the physiology of both of them. The thinking that people used to assign CBT a reasonable prior plausibility stems from a misunderstanding of what prior plausibility means. CBT is a placebo (according to my definition: an effective treatment with effects not due to a pharmacological effect, surgical effect, or other physical effects but mediated through communication).

    Placebos can be effective. Psychotherapy is quite effective. Psychotherapy is a placebo because the effects are not chemical or surgical. It turns out that placebos do tend to be effective for CFS because the underlying pathophysiology of CFS is low nitric oxide, and the primary therapeutic effect of a placebo is to raise nitric oxide by neurogenic means.

    There are two usual mistakes that people make in considering CFS and CBT together. The first is that there is “nothing wrong” in CFS so CBT is a reasonable approach for a disorder that has no physiological basis. This is not correct. CFS has many well documented symptoms that are all completely consistent with low NO, and the main symptom, exertional fatigue is characteristic of insufficient mitochondria and more rapid ATP depletion in muscle has been demonstrated in CFS. Mitochondria biogenesis is regulated by NO, so low NO leads to insufficient mitochondria which will always cause CFS.

    The second mistake is that if people with CFS get better with CBT, that means that there is no physiological basis for CFS. This is not correct.

    The results of the study you linked to do not surprise me. Increased physical activity is not necessary to increase mitochondria levels. What is necessary is increased NO levels. Increase NO levels and there will be increased mitochondria biogenesis and a resolution of the CFS.

    The authors make the mistake that most exercise physiologists make. It is not exercise that increases aerobic endurance, it is the compensatory mechanisms triggered by exercise that produce the increased mitochondria biogenesis which results in the increased aerobic endurance.

  70. Dr Benway says:

    There is a difference between experts defined by credentials, and experts defined by results of tests of performance. By testing people who have research credentials for their ability to correctly predict the outcomes of experiments, promoting those who pass the initial test, and continued monitoring of their performance if they are promoted to the panel of experts, the process can be made more objective.

    Expert opinion is useful when you lack facts, and the method you describe will weed out the more annoying Dunning-Krugers, so yay for that.

    I promise to shut up if you concede that this method is awesome in comparison to dictionary fishing or consulting the stars, but less awesome than objective measures of variables of interest, and that we ought to think long and hard before we forgo the pursuit of objective measures in favor of the “this expert might be better than flipping a coin” test.

  71. Dr Benway says:

    Oh rats, JMB, I forgot to address your point: the test to qualify the experts is objective. Yes, as you describe it, it is.

    But the expert’s opinion itself is subjective.

  72. JMB says:

    I would agree that objective measures of prior probability are better than expert opinion, even when expert subjectivity is monitored continuously by outcomes. However, there are significant gaps in our knowledge base of objective measures of prior probability. In those decisions in which we lack hardcore data (such as RCT type data) qualified expert opinion (SBM) performs better than burying your head in the sand (EBM). So expert opinion qualified by testing and monitored for reliability just fills in the gaps, but does so better than appointing a panel of experts based on past credentials.

    An expert’s subjective opinion can be measured objectively in a test. The difference in use of an expert opinion and a machine generated number, is that we must routinely calibrate the machine, but we must monitor the outcomes of the expert opinion.

  73. JMB says:

    In other words, I concede, but that is no reason for you to shut up. I enjoy reading your comments.

  74. BillyJoe says:

    “What is necessary is increased NO levels.”

    OH NO, NOT NO AGAIN!

  75. daedalus2u says:

    The issue of prior plausibility is that we don’t know how to calculate it. But we do know that what ever “prior plausibility calculator” we use should never tell us that a true result is not plausible because every true result must be by definition plausible. Every true result is always consistent with every piece of data, basic research as well as RCTs. If we are using a “prior plausibility calculator” that tells us a true result is implausible, that is a fault of our “prior plausibility calculator” and not a fault of the true result, or how the true result is presented.

    This is why the idea of prior plausibility should always be framed as a lack of prior implausibility. This is difficult for people to do because it is always easy to ignore data that doesn’t fit a hypothesis, or to stop looking for data if it is a hypothesis that you don’t like and then claim the proponents of the hypothesis have cherry-picked only the data that fits the hypothesis.

    Your default has to be “I don’t know” if a particular hypothesis is plausible or not until you have spent sufficient time to understand the hypothesis and the data behind it and are able to articulate why it is implausible. Only data is acceptable for determining if a hypothesis is implausible because it is only data which must be consistent with reality.

  76. geo says:

    Hi daedalus2u:

    This is getting ever more complicated by peripheral issues… curse these real life examples.

    I think that if CBT were providing useful and practical information that led to dramatic reductions in levels of disability, then this would be quite different to if CBT were simply altering the language with which patients chose to describe their level of disability and the way they chose to fill in questionnaires. You would class both of these as a placebo affect, but I think that the moral implications either has for the way in which CBT is promoted to patients would be rather different.

    I’m afraid that I’ve no idea about NO levels, and tend to assume CFS is unlikely to have any one cause.

  77. daedalus2u says:

    The mechanism by which CBT helps with CFS is neither of those. It can be real help. It is primarily a stress relief kind of effect. The individual has social interactions in a safe environment with a trusted individual and this reprograms the ANS to generate less superoxide and more NO. It is a relaxation response type of effect, a turning off of the “fight or flight” state.

    The conditions that CBT helps are all made worse by stress. Stress is a low NO state and the adverse effects of stress are mediated by low NO. The “beneficial” effects of stress are too, but the beneficial effects of stress are only beneficial in the short term. The major “benefit” is the increasing of metabolic resources by turning off non-essential systems (systems that are non-essential in the very short term). In the long term there are no non-essential systems, so in the long term high stress is always bad.

    I think most people promoting CBT don’t understand how it works. You don’t need to understand how it works for it to work. If you go into it expecting it to not work, it probably won’t. Since most explanations of CBT are probably wrong (I think that only the one I have given is correct), it is not irrational to think that the explanation the therapist is giving is bogus (because it is) and that the treatment won’t work (because the rational is bogus), which then becomes a self-fulfilling prophesy (because for any talk therapy to be effective there needs to be trust which turns off the “fight or flight” state).

    I think the same is true of pretty much all placebos. I put the placebo of talk therapy in a different class than the placebos of CAM because (most) talk therapists are not fooling themselves and their clients. The important part of talk therapy is the relationship and the trust between the clinician and patient. Usually that trust can’t happen if either the clinician or patient is lying.

  78. geo says:

    Hi Daedalus2u:

    “The mechanism by which CBT helps with CFS is neither of those… this reprograms the ANS to generate less superoxide and more NO. ”

    This is part of the problem I have with the emphasis on plausibility and science, rather than evidence based medicine.

    I worry that it leaves more room for people to think they know more than they do. Given the confused state of current CFS research I don’t think we can be so so sure of what is actually occurring, or by what mechanism.

    While I understand that there are extreme cases, like homeopathy, where our understanding of reality is such that we can be justifiably confident, prior to RCTs, that homeopathy is not effective – but I think that it is hard to restrain that faith in our own knowledge to only the most extreme of examples. People seem drawn to believing they know more than they do, and to having too much confidence in their judgements about how to best treat those they have authority over. I value science because it can be such an effective system for restraining our instinctive beliefs and prejudices… but it struggles to work effectively when the evidence is as lacking and ambiguous as seems to be the case with CFS – and a number of other medical conditions.

    Anyway, thanks for sharing your thoughts.

  79. daedalus2u says:

    geo, I understand where you are coming from, but there isn’t another way to do it. Prior plausibility has to be tied back to data. If you can’t do that, then you are not calculating a “prior plausibility”. The more data the calculation is tied to, the better the estimate of plausibility is.

    It is very easy for people to think they know more than they do, and to denigrate what other people know. At Mark Crislip’s other blog (rubor, dolor, calor, tumor, 11/23/10) he has a nice discussion of that. Incompetent people think they know more then they do and think that everyone else knows less, competent people know there is lots they don’t know but think that everyone else knows as much as they do.

    There isn’t a short-cut. If you want to estimate a prior plausibility you have to really understand the science involved. If you don’t understand the science and can’t tie that science back to data, then you are just guessing. Sometimes people guess correctly (as in CBT and CFS) but sometimes they guess wrong, as in the many treatments that fail in clinical trials. Sometimes a treatment is effective, but not for the reasons originally suggested, I think the health effects of statins are not due to inhibition of cholesterol synthesis.

    The strength of SBM is that it continues to apply all data to the problem. If there is a mismatch between basic science and clinical trials, SBM keeps plugging away at the science until they do match. EBM just says do more clinical trials.

  80. JMB says:

    @geo

    I don’t think the editors of this site disagree with the ordering of the hierarchy of evidence promoted by EBM. They just disagree about the relative weight given to each level in the hierarchy by EBM. So the weight of the lowest level of evidence is increased slightly, and the weight of the highest levels of evidence are modified (and often reduced) by prior probability. The highest level of evidence that is clinically feasible, is still required for justification of use in the healthcare. Conclusions that we need further research after meta analysis of clinical research into scientifically implausible mechanisms are not justified. Well designed RCTs can give us incorrect answers for implausible interventions based on the arguments by Ioannidis.

    It is the editors of this site that define what SBM is, not those of us that make frequent comments. daedulus promotes nitrous oxide as a scientific mechanism. It is an explanation for many poorly understood physiologic effects, often mediated by the autonomic nervous system (if I understand daedulus). That is a justification for research, not clinical use. I promote the old fashioned decision analysis, mathematical modeling, and Bayesian approach that came from computer aided diagnosis research. Computer aided diagnosis involved the translation of scientific mechanisms, clinical expertise, and evidence into medical decisions, with success measured by outcomes. To me, EBM is an oversimplification of what we learned in the past.

  81. daedalus2u says:

    JBM, nitric oxide, not nitrous oxide. Nitric oxide is NO and is a signaling molecule in many hundreds of pathways. Nitrous oxide is N2O, a not-so-great inhalation anesthetic (the mechanism of which is still not understood).

    I disagree that under SBM only treatments that have been put through clinical trials are suitable for clinical use. There was the famous example of parachutes. SBM would allow (require) parachute use even in the absence of clinical trials because the prior plausibility is so high.

    EBM wants to treat physiology as a “black box”, where everything going on inside the black box is inscrutable. SBM wants to scrape off the covering of the black box, look inside and try to figure out what is really going on. SBM wants everything to be transparent, EBM assumes everything except RCTs are completely opaque and then uses blinders to not look at it.

    SBM is more powerful because it draws on more information. But with greater power comes greater responsibility. SBM is easier to misuse because it does allow more information to be considered. If that information is cherry-picked or distorted, wrong prior plausibilities can be wrongly “calculated”. We shouldn’t not use more powerful techniques simply because they can be misused. EBM can be misused too.

    There isn’t a formulaic response to complex questions with limited data. EBM tries to force a formulaic response by limiting the data to only that generated utilizing RTCs. SBM adds everything else that is known to the RTC data, so SBM has the capacity to get better answers. But with more data come more degrees of freedom if only subsets of the data are looked at (actually more data reduces the degrees of freedom by adding more constraints).

    If you only have the capacity to look at 100 pieces of data, using only the most reliable 100 pieces of data will give you a better answer. That is the approach that EBM takes, truncate the data to only RCTs. SBM says, lets look at 1,000 or 10,000 pieces of data, lets look at all the data that could possibly be important. If you do look at all the data, then you will get a better answer. The problem is when do you stop looking at data. If you stop when the prior plausibility exceeds your cutoff, i.e. when it reaches 0.5000001%, then you have cherry-picked the data and are not doing SBM.

    Clinical trials under EBM are only possible when physiology is treated as a black box, when the patients in the clinical trial are all “equivalent”. If the patients are not equivalent, for example if you have a complete genome sequence of each patient, EBM can’t treat a set of those patients as being equal because they are not equal. More information (the genome sequence of each patient in the trial) turns the EBM RCT into a collection of anecdotes (because each patient is now unique).

    As scientists we know that more data is always better, but EBM can’t handle more data.

  82. JMB says:

    @daedulus thanks for the correction. NO not N2O.

    I personally have participated in an autopsy of a low altitude ejection. That is a clinical observation, not a clinical trial. Clinical observation is common sense with a trained eye. Well documented scientific mechanisms still need some input from clinical research, whether clinical observation or trials. Clinical observations of the rate of injuries from old style parachutes gave incentive to improve parachutes. They did not need a randomized prospective clinical trial to determine the need for replacing the old parachutes.

    I think you are right about the error of EBM treating underlying physiologic mechanisms as a black box. Errors in research occur not only from low prior probability, but from mistakes in experimental design failing to adequately isolate the part of the physiologic mechanism that we wish to test.

  83. geo says:

    re “It is the editors of this site that define what SBM is, not those of us that make frequent comments”:

    Wallace Sampson has posted here a few times on CFS, saying stuff like:

    “To those who still must argue that CFS and related disorders are diseases and not somatiform illnesses, and to patients who deny their problems’ origins, allow me to drop the fomalities of physician behavior and to speak in language used here.

    “You are wrong. Stop looking for information that fits your conceptions, and try to learn what is happening to you; how your life – the only one you may have – may slip away under a pall of unhappiness and contrariness, while disallowing entry to sources that may help you the most. There is hope if defenses relent, and none if they are maintained. ”

    Just as I don’t think the evidence is there to yet support daedalus2u’s claims, I don’t think it is there for Sampson’s either – and Sampson’s also seem more likely to lead to patients being treated unfairly. Other contributor’s have not condemned Sampson’s writings – indeed, Harriet Hall defended his comments thus: “Dr. Sampson stuck his neck out and stated a strong opinion.” Surely a defence that could be used for all manner of quackery (Quackwatch definition: Quackery can be broadly defined as “anything involving overpromotion in the field of health” – pretty similar to sticking you neck out with a strong opinion imo).

    Given our tendency to exaggerate our own levels of knowledge, and to have a misplaced faith in those we see as experts, I think it’s better to aim for an undue scepticism of our own abilities – we’re shooting through a cross wind.

    I’m still not entirely clear on the differences between EBM and SBM – it seems like much can depend upon the preferences of those using the terms, and some people’s conceptions of EBM would be identical to some people’s understanding of SBM. I think I’ve explained why some conceptions of SBM concern me, and that was all I intended to do. Thanks.

  84. daedalus2u says:

    geo, not to be too much of a noodge, but you have to actually do a pretty good look at the data that is there before you can say you don’t think there is enough.

  85. geo says:

    I understand that. I’m just reading a Maes and Twisk paper on ‘oxidative and nitrosative stress pathways’ (and reading back through the publication history) – I don’t think the evidence is there. I’m happy to be shown I’m wrong (although maybe that discussion would be rather OT here).

    Surely you don’t think the evidence is there for Sampson though – so you should be able to appreciate my point from that perspective.

  86. daedalus2u says:

    There is a lot of wrong stuff in the NO literature, especially as it relates to CFS. I have writen a blog about NO and CFS.

    http://www.chronicfatiguetreatments.com/wordpress/treatments/chronic-fatigue-syndrome-nitric-oxide/

    Only the blog and comments under my name are mine, there rest of the site isn’t.

  87. Dr Benway says:

    The Sampson quote, let it go dude. Holding onto hurt feelings for months on end isn’t good for you.

    Here, watch this video of the cutest parrot ever. You’ll feel better.

  88. daedalus2u says:

    In defense of Sampson (who I completely disagree with, CFS is an actual, real physiology-caused metabolic state), the only “treatment” is to raise NO levels and so “reset” physiology to a high NO state. There are no accepted pharmacological treatments that will do that, there are only placebos.

    If you have CFS, treating as if it is a somatiform disorder and trying to put yourself in a state where you are no longer “fighting” it, will help raise your NO level and will help in recovery. Treating it as something to “fight” will make it worse. Fighting with people who say it is somatiform will make it worse too. CFS is one of the things that trying to “fight” it makes it worse. Dr Benway is right, holding on to hurt feelings will make it worse too.

    My bacteria will do it, but there is nothing published on my bacteria that demonstrates that they will. I have a lot of theory, and a couple anecdotes (including a recovery from ARDS that the MDs involved called “miraculous” and had no explanation for (and wouldn’t listen to mine)).

  89. geo says:

    Thanks for the concerns, and I don’t mean to be personally condemning in my reply, but I think it’s socially dangerous to talk about our responses in such a pragmatic way. It undermines our sense of moral imperative and our commitment to believe what is most reasonable rather than that which we think will be most useful for us individually. One of my concerns about SBM is that it could encourage a faith in it’s own self-righting mechanisms that is seen to legitimise a reduction in our own individual moral responsibilities to speak out against such quackery.

    Anyway, I think I’ve made myself as clear as I can do on this one.

  90. qetzal says:

    deadalus2u:

    My bacteria will do it, but there is nothing published on my bacteria that demonstrates that they will.

    IMO, if you want to be taken seriously, you really need to stop making claims like this. Without robust clinical data, you don’t even know that raising NO levels will actually treat CFS, much less that your bacteria can do that.

    You think they’ll do it, based (as you say) on a lot of theory and a some anecdotes. But surely you know how many how many treatments have failed in the clinic, despite being supported by lots of theory and anecdotes.

    Claiming that your treatment method will work, and that it’s the only method that will work, is the hallmark of a woo-meister. If you don’t want to be taken for one, you should make an effort to distinguish what you know from what you hypothesize. It’s also good scientific practice. Reminding yourself that you don’t actually know that your idea is right helps you guard against your own bias and assumptions.

  91. JMB says:

    @geo

    One of my concerns about SBM is that it could encourage a faith in it’s own self-righting mechanisms that is seen to legitimise a reduction in our own individual moral responsibilities to speak out against such quackery.

    I don’t think you have read very many posts by the editor’s, and perhaps you should reread this post. The editors have previously reminded commentors that they have an obligation to speak out about quackery, even when those are soft targets (when some commentors have pointed out that the editors are shooting fish in a barrel.) Tolerance of the editors for opinions expressed in this forum relate to the ethics of an open forum, not the moral obligation to slam every quack statement. Tolerance of the editors for statements claiming to be scientific assertions by commentors may result in a challenge by the editors (depending on how busy they are).

    Both EBM and SBM have faith in the self-righting mechanism of British empiricism. However, SBM recommends greater skepticism about results of RCTs (including skepticism based on the arguments of Ioannidis). I don’t think any of the editors would accept the opinion that SBM is less skeptical and more faith based than EBM. If anything, EBM puts blind faith in evidence obtained from randomized clinical trials with adequate controls and statistical power, even though in science skepticism is required of any evidence and experimental design.

  92. geo says:

    @ JMB – I should have been clearer. Wallace Sampson is one of SBM’s contributors, not just a poster in the comments section.

    I’m certainly not saying that those at SBM would be slow to criticise those they see as not being a part of the SBM in-crowd… but what about those who are?

    The evidence supporting Wallace’s views does not seem terribly compelling, but he has escaped censure partly because he is working withing the social frame-work of SBM and because his views fit with the prior beliefs many have about CFS. The blindness of the faith in EBM is something that appeals to me, as I think it will limit the overall level of trust and faith that seems to expand were we are able to be selective over its purview. Everyone wants their guys trusted, while the claims of opponents are treated dismissively – restricting our ability to make that distinction seems a good way of encouraging greater levels of scepticism.

  93. pmoran says:

    Geo: The evidence supporting Wallace’s views does not seem terribly compelling, but he has escaped censure partly because he is working withing the social frame-work of SBM and because his views fit with the prior beliefs many have about CFS.

    I agree with much that you say. I may be the only other to point out loud to weaknesses in the chains of evidence for daedalus2′s wide-reaching hypotheses on NO physiology. Yet he does make very valuable contributions here.

    There IS considerable evidence favouring Wally’s views on CFS, at least regarding many of the patients so labelled. More importantly, on present knowledge, his approach may offer the best chance for most CFS sufferers to get back to a normal life, whatever initially caused the illness.

    Hoping that one day other theories will reach fruition, validate the sufferer’s persistence in disease mode, and yield some pill that will magically terminate the condition without effort may condemn many to unnecessary wastage of a life, even if others of a possibly polyglot population will eventually be helped by the making of a precise pathophysiological diagnosis. They will then assume a different diagnosis.

    This is a field that I only follow somewhat peripherally, so I am quite prepared to be shot down, so long as those pointing to this or that of the numerous abnormalities supposedly shown in CFS patients, have an answer to “chicken or egg” questions i.e. is this a primary cause of the condition, or the result of chronic inactivity prompted by other causes?

  94. geo says:

    “There IS considerable evidence favouring Wally’s views on CFS, at least regarding many of the patients so labelled. More importantly, on present knowledge, his approach may offer the best chance for most CFS sufferers to get back to a normal life, whatever initially caused the illness.”

    I’d really be interested to see what evidence. Whenever I look in to CFS papers I’m amazed by how weak they are (those written from a primarily psychological or biological perspective). This is a recent one that really shows nothing once you’ve looked in to the questionnaires used and references cited: http://content.karger.com/ProdukteDB/produkte.asp?Aktion=ShowFulltext&ArtikelNr=319312&Ausgabe=254424&ProduktNr=223864#AC

    A collection of such papers can be combined for a meta-analysis – and then you’ve got what could be considered ‘considerable evidence’, but it’s all built upon nothing.

    Lets get the Wallace quote again, before we examine it on pragmatic grounds:

    “To those who still must argue that CFS and related disorders are diseases and not somatiform illnesses, and to patients who deny their problems’ origins, allow me to drop the fomalities of physician behavior and to speak in language used here.

    “You are wrong. Stop looking for information that fits your conceptions, and try to learn what is happening to you; how your life – the only one you may have – may slip away under a pall of unhappiness and contrariness, while disallowing entry to sources that may help you the most. There is hope if defenses relent, and none if they are maintained. ”

    Even if it’s not true that CFS is a somatiform illness, this would be acceptable because such an approach might offer patients the best chance to get back to a normal life?

    This is so far from the way I think medicine should be done that it’s difficult to explain exactly why I think you wrong. Clearly Wallace’s claims fall within Quackwatch’s definition of quackery – I think this is bad. To make claims to patients without having clear evidence that they are true serves to undermine the trust that a good doctor-patient relationship requires (never mind that, ignoring the pragmatic considerations, it’s just wrong). Many CFS patients no-longer trust mainstream medicine because of quackery like Wallace’s. Even if you don’t know much about science or medicine, you just need to be of normal intelligence and have the vaguest of ideas about the state of CFS research, and how it is diagnosed, to realise that anyone making the sort of broad claims Wallace does is not to be trusted. To have those claims presented as being Science Based Medicine surely cannot be a good thing.

    I’m so baffled by how anyone could try to defend Wallace’s comments that I’m not entirely sure if I’ve countered your arguments or not. Are you saying that we should treat all patients with CFS as if they’re suffering from a somatiform disorder, even though this is unlikely to be true, because otherwise they’ll waste too much of their time wishing for an honest explanation of their illness?

    This rather pointless paper examines the views of CFS patients and medical staff, and included the quote: “We do need to be able to trust the people who treat us”. I don’t think that’s an unreasonable desire. http://www.biomedcentral.com/content/pdf/1471-2296-11-89.pdf

    Have you read much from patients who have been treated as if their medical condition were psycho-somatic, only for a non-psychological cause to later be found? From what I’ve seen, they don’t tend to be grateful for having been saved the trouble of getting caught up in a sickness role. Living with mental health problems is incredibly difficult. Being treated as if you are mentally ill when you are not seems to scar people rather badly too.

  95. daedalus2u says:

    In SBM, censure is reserved for those who hold and advocate views when there is clear and compelling data to the contrary. When the data is not so clear (as in CFS and how to treat it), not so much ;). Vigorous debate is to be encouraged so long as people can disagree without becoming disagreeable.

    I actually agree with Dr Sampson that the best course of treatment for a patient with CFS to proceed on right now, is to treat it as if it is a somatiform disorder with stress reduction and moderate exercise. Right now there isn’t a better treatment available. CFS patients should not wait until my NO bacteria become available. Even when they are available, my bacteria will work best along with the same treatments as for somatiform disorders, stress relief and moderate exercise.

    It is unfortunate that there is neuronal remodeling that often accompanies CFS that is the perpetuation of the idea (which at first is actually correct) that exercise will cause pain and so exercise should not be done. When there are insufficient mitochondria, this is a protective mind-set. Trying to use ATP that one does not have the capacity to produce can cause serious damage because muscles can be worked to death, until they necrose. Immune system activation induces feelings of fatigue so that muscles are not used when they do not have the ATP generation capacity to support their activity. Humans generate all kinds of rationalizations to justify why they feel they cannot exercise. The rationalizations may not be correct and may persist after the need for them has passed.

    Dealing with formerly adaptive ways of coping that have become maladaptive is extremely difficult. I think virtually all mental health issues fall into this category. Virtually all of them are adaptive in the short term (sometimes very short term, minutes) but maladaptive in the long term. Getting the regulation of everything back on track can be difficult because of the different time constants involved, the limited handles that can be manipulated, and the hysteresis of physiology. Changing feelings and behaviors requires changing neuroanatomy and for some types of change that is pretty slow.

    I think the wanting to wait until there is a “cure” is partly due to the neuronal remodeling that CFS produces. In “the wild”, if you had CFS and could wait, eventually it would resolve itself (because in “the wild” everyone had a biofilm of my bacteria). Waiting doesn’t spontaneously work now because the biofilm has been lost through bathing. But overdoing exercise is very easy for people with CFS to do. They need to be very careful and calibrate the amount of pain they can push themselves to endure without causing bad side effects (more muscle damage than can resolve overnight). This level is idiosyncratic.

    The low NO hypothesis of CFS does predict that there will be increases in neuropsychiatric disorders among those with CFS compared with non-CFS controls (because it is low NO that also causes neuropsychiatric disorders (I am working on writing this up)). But CFS is not caused by any type of mental health problem, both CFS and the various mental health problems are caused by the same thing, low NO.

    PM, in CFS, the mitochondria depletion precedes the exercise intolerance. In the study on CBT that geo linked to above, the exercise intolerance of CFS resolved with CBT without an increase in physical activity. It is not physical activity per se that results in increases in exercise tolerance, it is the compensatory pathways triggered by physical activity that produce the mitochondria biogenesis that results in the increased capacity for aerobic ATP production that is necessary for exercise tolerance. The level of activity that produces a certain level of mitochondria biogenesis depends on the basal NO level. NO triggers mitochondria biogenesis, higher basal NO means that less NO generation by physical activity is necessary to trigger mitochondria biogenesis.

  96. pmoran says:

    PM, in CFS, the mitochondria depletion precedes the exercise intolerance.”

    And you know this, how?

  97. pmoran says:

    Geo, Wally would probably not talk directly to any patient that way, either, but I was mainly reacting to your implication that there is no good evidence for Wally’s position. The mere fact that many CFS sufferers can recover through simple interventions indicates that at least this sample of that population has no enduring underlying abnormality of their immune systems, mitochondria, or whatever.

    I think patients would, or should, be approached on an individual basis. One can accept that the symptoms are real, as they are merely groos exaggerations of symptoms that are extremely prevalent in the supposedly “normal” population.

  98. geo says:

    @ daedalus2u: Sorry for side-stepping most of your reply (which I’m afraid I disagree with much of), but I really am more interested in the abstract philosophical side.

    “In SBM, censure is reserved for those who hold and advocate views when there is clear and compelling data to the contrary. When the data is not so clear (as in CFS and how to treat it), not so much ;) .”

    When did the evidence to the contrary become so clear and compelling that it was no longer acceptable to promote the notion that black people were more closely related to monkeys than white? What evidence was it that excluded this sort of scientific racism from Science Based Medicine?

    I don’t think that quackery was ever a legitimate part of science, and I think that the weakness of the evidence used by it’s proponents combined with the moral and social setting in which they were operating meant that they should have been censured from the start.

    People should be free to believe what they want. Certainly, researchers need to be free to develop hypotheses, gather evidence and test their predictions; but I think that to promote your beliefs as facts in a forum like SBM you should already have clear and compelling data.

  99. geo says:

    @ pmoran… It seems like you’re defending a reasonable position, and trying to presume that this is what Wallace meant to say. If you look at what he actually wrote, then their meaning is quite clear.

    This is one of the things that worries me. You wouldn’t be trying to so generously re-interpret Wallace’s words if you did not already think that he was one of the good guys. These sorts of cognitive biases are ever present, and I think that the systems of knowledge which work best are those that leave the least room for them.

    Anyway, it seems like I’ve gone full circle, and I can end with the quote I used in my first post. How poetic:

    “Science is the organized skepticism in the reliability of expert opinion.” – Richard Feynman.

  100. daedalus2u says:

    PM, Because there is considerable research demonstrating it.

    CFS often occurs acutely just after severe immune system activation, as in septic shock. Sepsis does cause mitochondrial depletion. Mitochondria are the only source of aerobic ATP. ATP depletion accompanies muscle fatigue (as measured in vivo by MRS). In CFS there is faster and more severe ATP depletion that takes longer to resolve than in non-CFS controls. There is measured mitochondrial depletion in CFS patients. Here is quite a nice paper on it.

    http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2680051/

    The fatigue of dilative cardiomyopathy and of COPD is similar to that of CFS in that it affects the mitochondria and muscle endurance and not muscle strength (so much). The fatigue of COPD and dilative cardiomyopathy is due to insufficient mitochondria in muscle and not due to insufficient O2 delivery.

    In normal controls it is heart output that limits VO2max. In CFS, DCM, COPD it is not heart output that limits VO2max. I am pretty sure that the fatigue of CFS, DCM, and COPD are all pretty similar and variations of similar pathways that reduce mitochondrial biogenesis.

    The anaerobic threshold is reduced in CFS, that is the transition to glycolysis occurs at a lower work output. The respiratory exchange ration (CO2 produced/O2 consumed) is higher in CFS patients (indicating greater oxidation of carbohydrate than fat) and increases faster with exercise than in normal controls. The pH in CFS skeletal muscle drops faster with exercise (indicating more glycolysis sooner) then in controls.

Comments are closed.