Nov 12 2010
Of SBM and EBM Redux. Part I: Does EBM Undervalue Basic Science and Overvalue RCTs?
During the most recent kerfuffle about whether or not Evidence-Based Medicine can legitimately claim to be science-based medicine, it became clear to me that a whole, new round of discussion and documentation is necessary. This is frustrating because I’ve already done it several times, most recently less than a year ago. Moreover, I’ve provided a table of links to the whole series at the bottom of each post*…Never mind, here goes, and I hope this will be the last time it is necessary because I’ll try to make this the “go to” series of posts for any future such confusions.
The points made in this series, most of which link to posts in which I originally made them, are in response to arguments from statistician Steve Simon, whose essay, Is there something better than Evidence Based Medicine out there?, was the topic of Dr. Gorski’s rebuttal on Monday of this week, and also from several of the comments following that rebuttal. Mr. Simon has since revised his original essay to an extent, which I’ll take into account. I’ll frame this as a series of assertions by those who doubt that EBM is deficient in the ways that we at SBM have argued, followed in each case by my response.
First, a disclaimer: I don’t mean to gang up on Mr. Simon personally; others hold opinions similar to his, but his essay just happens to be a convenient starting point for this discussion. FWIW, prior to this week I perused a bit of his blog, after having read one of his comments here, and found it to be well written and informative.
What’s in a Name?
One of Mr. Simon’s objections, in his revision, is this:
What is SBM? Here’s a definition found on the opening entry in the SBM blog:
“the use of the best scientific evidence available, in the light of our cumulative scientific knowledge from all relevant disciplines, in evaluating health claims, practices, and products.” http://www.sciencebasedmedicine.org/?p=1
But how does this differ from David Sackett’s definition of EBM?
“the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.” http://www.bmj.com/content/312/7023/71.full
The only substantial difference I see is the adjective “scientific” that appears twice in the definition of SBM. The claim on the SBM blog is that EBM ignores scientific plausibility. Actually, ignores is too strong a word.
“EBM ‘levels of evidence’ hierarchy renders each entry sufficient to trump those below it. Thus a ‘positive’ clinical trial is given more weight than ‘physiology, bench research or “first principles”,’ even when the latter definitively refute the claim.” http://www.sciencebasedmedicine.org/?p=42
(I agree that “ignore” is too strong a word, but I didn’t actually write it that way, as Dr. Gorski pointed out and as I think Mr. Simon was acknowledging above.)
A difference between Sackett’s definition and ours is that by “current best evidence” Sackett means the results of RCTs. I realize that this assertion requires documentation, which will come below. A related issue is the definition of “science.” In common use the word has at least three, distinct meanings: 1. The scientific pursuit, including the collective institutions and individuals who “do” science; 2. The scientific method; 3. The body of knowledge that has emerged from that pursuit and method (I’ve called this “established knowledge”; Dr. Gorski has called it “settled science”).
I will argue that when EBM practitioners use the word “science,” they are overwhelmingly referring to a small subset of the second definition: RCTs conceived and interpreted by frequentist statistics. We at SBM use “science” to mean both definitions 2 and 3, as the phrase “cumulative scientific knowledge from all relevant disciplines” should make clear. That is the important distinction between SBM and EBM. “Settled science” refutes many highly implausible medical claims—that’s why they can be judged highly implausible. EBM, as we’ve shown and will show again here, mostly fails to acknowledge this fact.
Finally, Mr. Simon has misinterpreted our goal at SBM:
But if someone wants to point out that EBM needs work, I’m fine with that. I dislike that they think that EBM needs to be replaced with something better.
You see EBM as being wrong often enough that you see value in creating a new label, SBM. I see SBM as being that portion of EBM that is being done thoughtfully and carefully, and don’t see the need for a new label.
I generally bristle when people want to create a new and improved version of EBM and then give it a new label.
I am as harshly critical of the hierarchy of evidence as anyone. I see this as something that will self-correct over time, and I see people within EBM working both formally and informally to replace the rigid hierarchy with something that places each research study in context. I’m staying with EBM because I believe that people who practice EBM thoughtfully do consider mechanisms carefully. That includes the Cochrane Collaboration.
Mr. Simon, we agree! Yes, we are pointing out that EBM needs work. Yes, SBM is that (tiny) portion of EBM that is being done thoughtfully and carefully, and if it were mainly done that way there would be no need to call attention to the point. Our goal is not to change the name of EBM (“give it a new label”). Our goal is to convince EBM to live up to its current name. Yes, it may self-correct over time, but we are trying to shorten that time. Bad things have unnecessarily happened, in part due to EBM’s scientific blind spot: As currently practiced, it doesn’t rationally consider all the evidence. We don’t see much evidence that people at the highest levels of EBM, eg, Sackett’s Center for EBM or Cochrane, are “working both formally and informally to replace the rigid hierarchy with something that places each research study in context.”
We chose to call our blog “science-based medicine” only because the term “evidence-based medicine” had already been taken, and we needed to distinguish ourselves from the inaccurate use of the word “evidence” in “EBM.” I’ve written about this before, and have made the point utterly clear:
These are the reasons that we call our blog “Science-Based Medicine.” It is not that we are opposed to EBM, nor is it that we believe EBM and SBM to be mutually exclusive. On the contrary: EBM is currently a subset of SBM, because EBM by itself is incomplete. We eagerly await the time that EBM considers all the evidence and will have finally earned its name. When that happens, the two terms will be interchangeable.
Plausibility Misinterpreted
Mr. Simon’s interpretation of our view of plausibility, like that of many others, is wrong:
I would argue further that it is a form of methodolatry to insist on a plausible scientific mechanism as a pre-requisite for ANY research for a medical intervention. It should be a strong consideration, but we need to remember that many medical discoveries preceded the identification of a plausible scientific mechanism.
I think, from his revision, that Mr. Simon understood Dr. Gorski’s explanation of why this was wrong, but I’m not certain. The misrepresentation of scientific plausibility is an issue that I’ve faced for years, as explained previously here:
Plausibility ≠ Knowing the Mechanism
Let’s first dispense with a simple misunderstanding: We, by which I mean We Supreme Arbiters of Plausibility (We SAPs) here at SBM, do not require knowing the mechanism of some putative effect in order to deem it plausible. This seems so obvious that it ought not be necessary to repeat it over and over again, and yet the topic can’t be broached without some nebbishy South Park do-gooder chanting a litany of “just because you don’t know how it works doesn’t mean it can’t work,” as if that were a compelling or even relevant rebuttal. Let’s get this straight once and for all: IT ISN’T.
Steve Novella explained why at the Yale conference and again here. We talked about it at TAM7 last summer. For a particularly annoying example, read the three paragraphs beginning with “Mr. Gagnier’s understanding of biological plausibility” here.
OK, I’ll admit that I’m beginning to learn something from such frustration. Perhaps we’ve not been so good at explaining what we mean by plausibility. The point is not that we don’t know a particular mechanism for homeopathy, for example; the point is that any proposed mechanism would necessarily violate scientific principles that rest on far more solid ground than any number of equivocal, bias-and-error-prone clinical trials could hope to overturn. The same is true for “energy medicine” and for claims based on non-existent anatomical structures (iridology, reflexology, auricular acupuncture, meridians, chiropractic “subluxations”), non-existent physiologic functions (“craniosacral rhythms“), or non-existent anatomic-physiologic relations (“neurocranial restructuring,” “detoxification” with coffee enemas, dissolving tumors with orally administered pancreatic enzymes). The spectrum of implausible health claims euphemistically dubbed “CAM” is full of such nonsense.
Reader daedalus2u proposed a useful way to clarify the point:
I think the idea of prior plausibility should actually be reframed into one of a lack of prior implausibility. It isn’t that one should have reasons to positively think that something is plausible before testing it, but rather that one should not be able to come up with reasons (actually data) why it is fatally implausible.
Some of what We deem implausible will not be fatally so, of course. Implausibility can be based not only on established physical and biological knowledge, but also on studies, as is the case for sticking needles into people, injecting them with chelating agents, or claiming that autism is caused by childhood immunizations.
EBM, Basic Science, and RCTs
Steve Simon wrote, “I have not seen any serious evidence of EBM relying exclusively on RCTs. That’s certainly not what David Sackett was proposing in the 1996 BMJ editorial…” And: “No thoughtful practitioner of EBM, to my knowledge, has suggested that EBM ignore scientific mechanisms.”
Want serious evidence? Consider these quotations from Cochrane reviews, originally posted here:
In view of the absence of evidence it is not possible to comment on the use of homeopathy in treating dementia.
There is not enough evidence to reliably assess the possible role of homeopathy in asthma. As well as randomised trials, there is a need for observational data to document the different methods of homeopathic prescribing and how patients respond.
There is currently little evidence for the efficacy of homeopathy for the treatment of ADHD. Development of optimal treatment protocols is recommended prior to further randomised controlled trials being undertaken.
Though promising, the data were not strong enough to make a general recommendation to use Oscillococcinum for first-line treatment of influenza and influenza-like syndromes. Further research is warranted but the required sample sizes are large.
Yes, EBM undervalues basic science and overvalues RCTs when the former is sufficient to reject a claim. EBM also undervalues experimental evidence other than RCTs when the former is sufficient to reject a claim, as will be discussed. Here is how a truly evidence-based review might conclude a discussion of homeopathy for dementia:
The probability that homeopathy is specifically therapeutic for dementia is, for all practical purposes, zero.
The following is from my first post on the topic, in which I reviewed the overwhelming evidence—from basic science and pre-clinical research—that homeopathic ‘remedies’ have no, specific therapeutic actions, and wondered why the most esteemed exponents of EBM have written that such treatments are “promising” and that “further randomized trials are needed.” I included the Center for Evidence-based Medicine’s formal “Levels of Evidence” scheme (not copied here), the pertinent quotation from Sackett’s 1996 editorial, my opinion that this failure of EBM was initially unintended, how Sackett et al eventually did address “CAM,” and the Cochrane abstracts quoted above:
It wasn’t meant to be like this. When I first discussed with my fellow bloggers the curious absence of established knowledge in the EBM “levels of evidence” hierarchy, at least one insisted that this could not be true, and in a sense he was correct. David Sackett and other innovators of EBM do include basic science in their discussions, but they recommend invoking it only when there are no clinical trials to consider:
Evidence based medicine is not restricted to randomised trials and meta-analyses. It involves tracking down the best external evidence with which to answer our clinical questions…And sometimes the evidence we need will come from the basic sciences such as genetics or immunology. It is when asking questions about therapy that we should try to avoid the non-experimental approaches, since these routinely lead to false positive conclusions about efficacy. Because the randomised trial, and especially the systematic review of several randomised trials, is so much more likely to inform us and so much less likely to mislead us, it has become the “gold standard” for judging whether a treatment does more good than harm.
That statement is consistent with EBM’s formal relegation of established knowledge to “level 5,” as seen in the Figure. I am not a historian of EBM and don’t care to be, but I suspect that the explanation for this choice is that “they never saw ‘CAM’ coming.” In other words, it probably didn’t occur to Sackett and other EBM pioneers that anyone would consider performing clinical trials of methods that couldn’t pass the muster of scientific plausibility. Their primary concern was to emphasize the insufficiency of basic science evidence in determining the safety and effectiveness of new treatments. In that they were quite correct, but trials of “CAM” have since reminded us that although established knowledge may be an insufficient basis for accepting a treatment claim, it is still a necessary one.
Take note: Sackett wrote, “we should try to avoid the non-experimental approaches, since these routinely lead to false positive conclusions about efficacy.” My point is that pre-RCT evidence does not routinely (if ever) lead to false negative conclusions. In that passage, moreover, Sackett seems to suggest that the only alternative to a “non-experimental approach” is an RCT; yet there are often other types of experiments that can definitively refute treatment claims, as will be discussed. Eventually Sackett et al did catch wind of “CAM,” but they got it exactly wrong:
Lacking that perspective, Sackett’s Center for Evidence-Based Medicine promulgates an “Introduction to evidence-based complementary medicine” by “CAM” researcher Andrew Vickers. There is not a mention of established knowledge in it, although there are references to several claims, including homeopathy, that are refuted by things that we already know. Vickers is also on the advisory board of the Cochrane CAM Field, along with Wayne Jonas and several other “CAM” enthusiasts.
In another post I cited the 2006 Cochrane Review of Laetrile:
A 2006 Cochrane Review of Laetrile for cancer would, if its recommendations were realized, stand the rationale for RCTs on its head:
The most informative way to understand whether Laetrile is of any use in the treatment of cancer, is to review clinical trials and scientific publications. Unfortunately no studies were found that met the inclusion criteria for this review.
Authors’ conclusions
The claim that Laetrile has beneficial effects for cancer patients is not supported by data from controlled clinical trials. This systematic review has clearly identified the need for randomised or controlled clinical trials assessing the effectiveness of Laetrile or amygdalin for cancer treatment.
Why does this stand the rationale for RCTs on its head? A definitive case series led by the Mayo Clinic in the early 1980s had overwhelmingly demonstrated, to the satisfaction of all reasonable physicians and biomedical scientists, that not only were the therapeutic claims for Laetrile baseless, but that the substance is dangerous. The subjects did so poorly that there would have been no room for a meaningful advantage in outcome with active treatment compared to placebo or standard treatment… The Mayo case series “closed the book on Laetrile,” the most expensive health fraud in American history at the time, only to have it reopened more than 20 years later by well-meaning Cochrane reviewers who seemed oblivious of the point of an RCT.
Is that review not serious evidence that the Cochrane Collaboration overvalues RCTs? In this case, moreover, it wasn’t only basic science that Cochrane ignored, but a definitive piece of clinical research that was not an RCT. Sure, I know that Cochrane is not the only pinnacle of EBM, but it’s one of them.
In both that post and another, I called attention to a statement that Edzard Ernst, the most prolific EBM-style “CAM” researcher of the past 20 years, had made in 2003:
A couple of years ago I was surprised to find that one of the authors of [the Cochrane Laetrile] review was Edzard Ernst, a high-powered academic who over the years has undergone a welcomed transition from cautious supporter to vocal critic of much “CAM” research and many “CAM” methods. He is now a valuable member of our new organization, the Institute for Science in Medicine, and we are very happy to have him. I believe that his belated conversion to healthy skepticism was due, in large part, to his allegiance to the formal tenets of EBM. I recommend a short debate published in 2003 in Dr. Ernst’s Focus on Alternative and Complementary Therapies (FACT), pitting Jacqueline’s countryman Cees Renckens against Dr. Ernst himself. Dr. Ernst responded to Dr. Renckens’s plea to apply science to “CAM” claims with this statement:
In the context of EBM, a priori plausibility has become less and less important. The aim of EBM is to establish whether a treatment works, not how it works or how plausible it is that it may work. The main tool for finding out is the RCT. It is obvious that the principles of EBM and those of a priori plausibility can, at times, clash, and they often clash spectacularly in the realm of CAM.
I’ve discussed that debate before on SBM, and I consider it exemplary of what is wrong with how EBM weighs the import of prior probability. Dr. Ernst, if you are reading this, I’d be interested to know whether your views have changed. I hope that you no longer believe that human subjects ought to be submitted to a randomized, controlled trial of Laetrile!
Uh, talk about “suggesting that EBM ignore scientific mechanisms”! When the principles of EBM and those of a priori plausibility clash spectacularly in the realm of CAM, it is a priori plausibility that should take precedence—not merely because the latter renders RCTs unnecessary, but because for such questions RCTs tend to confuse rather than clarify, as will be discussed further in the next part of this series.
I am happy to report that Dr. Ernst wrote me privately about that passage, with the answer that I’d mostly hoped for:
Have I changed my mind? I am not as sure as the sceptics seem to be that I ever was a supporter of CAM and I am still a bit sceptic about the sceptics [which perhaps makes me the "ueber-sceptic"]. Would I argue for more Laetrile studies? NO.
Even more to the point, perhaps, is a recent editorial by Dr. Ernst in which he calls homeopathy “absurd” and compares it to other, obvious absurdities, which I doubt he’d have done only a few years ago:
Should we keep an open mind about astrology, perpetual motion, alchemy, alien abduction, and sightings of Elvis Presley? No, and we are happy to confess that our minds have closed down on homeopathy in the same way.
This kind of clear thinking, as easy as it ought to be for intelligent people, seems oddly difficult for those steeped in EBM. I’ll offer another example in part 2, as part of my response to Mr. Simon’s assertion that “There is some societal value in testing therapies that are in wide use, even though there is no scientifically valid reason to believe that those therapies work.”
…
*The Prior Probability, Bayesian vs. Frequentist Inference, and EBM Series:
1. Homeopathy and Evidence-Based Medicine: Back to the Future Part V
2. Prior Probability: The Dirty Little Secret of “Evidence-Based Alternative Medicine”
3. Prior Probability: the Dirty Little Secret of “Evidence-Based Alternative Medicine”—Continued
4. Prior Probability: the Dirty Little Secret of “Evidence-Based Alternative Medicine”—Continued Again
5. Yes, Jacqueline: EBM ought to be Synonymous with SBM
6. The 2nd Yale Research Symposium on Complementary and Integrative Medicine. Part II
7. H. Pylori, Plausibility, and Greek Tragedy: the Quirky Case of Dr. John Lykoudis
10. Of SBM and EBM Redux. Part I: Does EBM Undervalue Basic Science and Overvalue RCTs?
11. Of SBM and EBM Redux. Part II: Is it a Good Idea to test Highly Implausible Health Claims?
12. Of SBM and EBM Redux. Part III: Parapsychology is the Role Model for “CAM” Research
13. Of SBM and EBM Redux. Part IV: More Cochrane and a little Bayes
14. Of SBM and EBM Redux. Part IV, Continued: More Cochrane and a little Bayes
15. Cochrane is Starting to ‘Get’ SBM!
16. What is Science?
142 Responses to “Of SBM and EBM Redux. Part I: Does EBM Undervalue Basic Science and Overvalue RCTs?”
Stolen from another of Kimball Atwood’s posts:
Although a reasonably high prior probability may not be a SUFFICIENT basis for incorporating a treatment into general use, it is a NECESSARY one.
Moreover, high prior probability is a NECESSARY basis for seriously considering such a treatment at all; that is, for both scientific and ethical reasons it is a PREREQUISITE for performing a randomized, controlled human trial.
A perfect and succinct summary I think.
Does it need to be more explicitly stated that you are saying RCTs are overvalued only when the results of preclinical science suggest low, or zero pre-test probability? (Apologies if that is not what you are saying) Otherwise, isn’t there a danger that quacks can claim that their brand of snake oil is good for health based on pre-clinical science even if it is not supported by RCTs or other clinical evidence, and justify this by saying that RCTs are overvalued, and that they are practising SBM?
I think it would be helpful to add a category “what is SBM” or something similar under your list at the top of the web page with this and related posts. You might have written about this before repeatedly, but it is not that easy to find on the website.
Yes, that is what I’m saying, have said repeatedly, and I don’t know how much more explicit I can get (see Billy Joe’s comment). Please read, in its entirety,
Yes, Jacqueline: EBM ought to be Synonymous with SBM
Regarding “what is SBM,” look here: http://www.sciencebasedmedicine.org/?p=1
Excellent point. Plausibility based on pre-clinical data fairly often results in suggested therapies that fail to pan out when tested in RCTs. In other words, preclinical plausibility often leads to false positives, but it rarely leads to false negatives.
” . . . isn’t there a danger that quacks can claim that their brand of snake oil is good for health based on pre-clinical science . . .”
They already do this, but under investigation their “theories” fall apart. The chiropractic “subluxation” and homeopathy are two examples.
Re: the idea that preclinical data rarely lead to false negatives. I agree with this, but I don’t think that anyone can really back this assertion up with numbers. The reason why is because if something tests negative in a pre-clinical setting, it will never be tested clinically, so there’s a bias present there.
A second point is that another way of framing the problem with EBM is that the hierarchy places small, low-quality RCTs above a massive, excellent-quality body of pre-clinical studies. In principle, I actually don’t have a problem with the idea that RCTs should be at the top of the hierarchy. If enough high-quality RCTs showed that homeopathy had a strong effect, then that really would mean that all of chemistry and physics were wrong. The problem is the idea that ANY RCT data trumps ANY basic science data, regardless of quality.
In a way, the problem is that EBM is like the internet, which was designed with the idea that everyone would play according to the rules, so there was no security built in, and it had to be added on later. EBM assumes that the only reason a clinical trial would be performed is because certain standards were met in terms of basic science data, case reports, unblinded studies, etc. The people who run RCTs for non-science-based “therapies” that don’t meet those standards are like spammers on the internet, who take advantage of holes in the system.
@ evilrobotxoxo:
I figured someone would raise that point. It presages Part II of this series, but in the meantime consider two scenarios. In the first, the claim is so unlikely, based on established knowledge, that no practical amount of clinical data can save it. Homeopathy, claims based on psychokinesis (eg, “distant healing,” “therapeutic touch,” “applied kinesiology”), claims based on other fanciful “energies” (craniosacral rhythms) or claims based on fanciful homunculi (iridology, reflexology, auricular acupuncture), for example. The reason that no amount of clinical data can save such claims is that clinical data are necessarily more messy and less voluminous than the data that established the claims as being hopelessly implausible in the first place. Nevertheless, these are being tested clinically, thanks to the NCCAM and other naive EBMers, and guess what the answers eventually, after many false positives and many years and much waste and more, are found to be?
The second scenario is one in which the claims are not hopelessly implausible, but for various reasons are usually substantially more implausible than most biomedical scientists are used to: injecting or feeding biologically active substances, touted by “CAM”sters, to cure cancer, atherosclerosis, and everything else, sticking needles into people for what ails them, cracking backs (or necks) to lower blood pressure or treat asthma. Again, the claims are being tested clinically, thanks to the NCCAM and others, and guess what the answers are? Yup. But only after the same useless exercise as described above, this time with the added feature of human subjects being tortured and injured.
Great, huh?
KA
@ Kimball
I’ve know you’ve made the point before that you do rely on RCTs and don’t devalue them relative to pre-clinical science when the pretest probability is high, but in this post the distinction is not clear (to me anyway). Billy Joes’ comment does not address the importance of RCTs vs preclinical science when interpreting evidence in that setting.
I realise you’ve repeatedly posted about what is SBM, but I have tried before to search this website to find the relevant posts and found it hard to find them, which is why I suggested a specific category. You’ve got categories for lots of other things. It seems a shame not to have one for SBM/EBM issues.
@ Jann Bellamy
But what about someone who claims that vitamin K or magnesium or whatever is good for bone health based on reasonable preclinical studies, even if it is not supported by clinical studies. The point I was making was that they can say their claims are founded in SBM because SBM says RCTs are overvalued, and cite this post. Of course it would be wrong, but I don’t see anything in this current post that would stop this misuse of the term SBM.
@Kimball:
I agree that there are multiple scenarios, and it can get complicated. However, I still think that a major part of the problem is that the current EBM framework presupposes plausibility based on basic science or preliminary clinical observation. I guess one could argue that CAM would claim to fall into the second category, e.g. if this many people claim to have positive results from acupuncture, then we should test it.
Interestingly, one of the most commonly-discussed topics on this blog is the vaccine-autism link, which is something that was actually quite plausible in the absence of clinical evidence.
I think Mark makes a good point that SBM is not clearly defined on the web site, unless one searches and reads through a bunch of posts to find it. Maybe there should be a “What is SBM?” link on the front page or something.
@ mark:
You are right, they could say that but they would be taking the “RCTs are overrated” statement totally out of context. I don’t think one could ever design an epistemology of medicine that couldn’t be twisted, misrepresented, cherry-picked or in some other manner corrupted by the pro-CAM crowd.
mark,
C’mon. This post says:
That is the same thing that my words quoted by Billy Joe say. In the “Yes, Jacqueline” article that I asked you to read, I wrote:
How much clearer can I be? There is absolutely nothing in my writing, or in the writing of any of my colleagues here on SBM, to suggest that
The language in this post, quoted above, “would stop this misuse of the term SBM.” Please excuse me if this seems insulting, but do you understand the meaning of “insufficient” and “necessary” in these passages?
@ Kimball
I’m sorry I’ll drop out of the comments here. I do understand what insufficient and necessary mean.
I have been a long time reader at SBM, but every time the whole SBM vs EBM comes up, I just don’t get the difference. After your last post, I decided to try and understand as best as I could. I re-read as many of your previous posts as I could find and Sackett’s BMJ editorial. I understand the issues highlighted about pre-test probability and implausibility, which seem to be relevant mainly to complementary medicine. But I really struggle to see any other differences. I won’t comment further.
I appreciate that you have repeatedly posted on all these issue, but each post is generally read in isolation. To me, the main message of this post is that EBM places too much weight on RCTs and not enough on basic science. As I posted above, I think that someone reading this post in isolation could make that claim and say it is supported by this post because it stresses giving more emphasis to basic science and less to RCTs with little qualification. The sentence you quote is easily missed in a very long post, and readers may not be familiar with your other posts.
I never suggested you or your colleagues would make such a statement.
From now on I’ll just skip the whole EBM vs SBM posts.
Hi mark I just chatted with you in another thread.
Dr. Atwood’s criticism of much (but not all) EBM is sound, in that prior plausibility has not been given sufficient consideration. This is not merely his opinion but is dictated by Bayes’ theorum.
Doctors and social scientists have gotten away with ignoring prior odds because the variable becomes moot in a research environment where its value remains similar across comperable studies. In such an environment frequentest statistics are good enough.
Oh, I forgot to say that “too much emphasis on RTC” verses “not enough emphasis on RTC” is a red herring. You have to ask, why do we value RTCs so much? Politics? Wat?
Really, RTCs are nothing more than an attempt to control for confounding variables. *shrugs*
Sadly, amazingly, incredibly, all evidence is important, whether of the RTC form or not. All relevant evidence relevant to a proposed hypothesis must be weighed if we want the highest level of confidence in our conclusion about that hypothesis.
And thus the CAMsters would reveal their tunnel vision.
Consider these two hypothesis:
1. Homeopathy is true.
2. Homeopathy is not true.
Let’s assign a probability to the first hypothesis and call it x. The second hypothesis must necessarily then have a probability of 1-x. Note that increasing the probability of either hypothesis decreases the probability of the other.
Now, everything currently understood about physics and chemistry stands in support of the second hypothesis and in contradiction to the first. So if we increase x on the basis of a few positive patient reports or a few surprising RTCs, we must be mindful of the fact that we are at the same time assigning a lower level of confidence to all that has been established within the basic sciences to date.
Only a very egocentric individual would feel entitled to pass judgment on the complex, peer reviewed literature published in physics and chemistry, simply on the basis of a few patients testimonies or the word of a family member who happens to sell Herbalife.
I DO agree that there are many many scenarios, and it can get complicated.
But, I still think that the key problem is that the current EBM framework presupposes plausibility based on basic science or preliminary clinical observation. I guess one could argue that CAM would claim to fall into the second category, for example. if this many people claim to have positive results from acupuncture, then we should test it.
I think that Dr Atwood has provided an excellent description of the errors in the formalism of EBM. Those errors in the formalism prevent low probabilities from being effectively used in a decision analysis to indicate the lack of benefit of CAM treatments. What good is a formalism that doesn’t allow refutation of a hypothesis with both low a priori and low a posteriori probability?
I would suggest another way to explain the difference between EBM and SBM; the recognition of the placebo effect. The formalism of EBM could be more complete (and closer to SBM) if it would include the principle that a scientifically acceptable mechanism of action must be determined for a treatment to not be labeled a placebo. By denigrating basic clinical science, EBM has lost one of the primary classifiers of placebo effect. Perhaps that is why EBM cannot declare homeopathy (and several other equally implausible CAM therapies) placebo effect.
As suggested in a comment by Always Curious, when a strong unexpected effect is noted in a clinical trial, it can become an incentive for basic clinical science research to explain a mechanism of action. There is no reason under EBM to evaluate an observed effect for mechanism of action.
mark,
Maybe this will read better for you:
Prior Probability is a NECESSARY, but generally not SUFFICIENT, basis for incorporating a treatment into general use. For that you also need Clinical Trials.
That puts the emphasis on both prior probability and clinical trials. It’s not a fight of one against the other.
A corollary: “RTCs to test hypotheses of very low prior probability will send our country on expensive wild goose chases, as any positive results will be wrong.”
Sorry I’m spamming with all these posts, but I’m searching for the best way to explain the importance of prior odds.
Analogy:
The San Diego PD arrive at a home after someone has been shot dead. They see a tall figure running from the back of the house. They give chase but lose him. Later while patrolling the neighborhood they see a guy who looks like the runner. They arrest him and send his DNA samples off to the lab. If the test comes back positive, the cops have their killer, correct?
Ok, same story as above. However, this time the cops don’t chase anyone. They simply pull a random name of someone living in China from a large database of everyone living on Earth. They get a tech in China to draw the guy’s blood and send it off to the lab.
Now if the Chinese guy’s DNA matches samples from the crime scene, will we lock him up for murder?
You look at the package insert for the DNA test and read an estimated false positive rate of about a million to one.
Wait, what? False positives can’t be the same for story one and story two. Story two *must* have a false positive rate of 100% or 1:1.
Consider in the decision to treat that EBM values basic science .01 and RCTs .99. In the decision to treat, SBM values basic science .1 and RCTs .9. I think those formulae are pretty basic simplifications of the process, but serve to get the point across. In the decision to treat, SBM does not test tube (or petri dish) results to being equal in importance to RCTs. Basic science findings do not justify treatment when there are negative results for benefits in clinical trials. In language descriptions we rely on ambiguous phrases (ambiguous in a mathematical sense) to convey the relative value of factors. Of course, Dr Atwood is really defining boundary (necessary) conditions for the decision rather than fuzzy estimates for the factors.
In the Bayes paradigm, the experimental design provides a likelihood ratio. That likelihood ratio multiplies the prior probability of the hypothesis to arrive at the posterior probability after the experimental result. There is limit to the power of an experimental design, a frequentist concept that I think relates to the limits on the likelihood ratio. When the a priori probability is less than .0001, it is hard to imagine any experiment having enough power to raise that probability to .1 in the posterior result. However, when the prior probability is .7, it is easy to imagine an experiment with enough power to result in a lower posterior probability of .1.
“In the decision to treat, SBM does not test tube (or petri dish) results to being equal in importance to RCTs.”
Sorry that should read, “In the decision to treat, SBM does not raise the value of test tube (or petri dish) results to being equal in importance to RCTs.”
Instead of saying negative results for benefits I should have said null results. Negative implies that the experimental arm fared worse than the control arm of a clinical trial.
I would agree that it will self-correct over time, but remember time can be a very precious resource for patients. Many can die waiting for self-correction. For a practicing doctor, it is unethical to wait for correction, it is better to slam the correction into place. It is also a waste of resources to wait for self correction. Personally, I think EBM has been so badly tainted by woo, that it is time to replace the moniker and start over again as SBM, incorporating the concepts put forth by the editors of this site.
That’s good, Imma translate it into grandma-speak:
I think the language below will confuse the n00bs:
This language implies that a political struggle exists between two parties. Big mistake! A mathematical “value” is not a political “value.”
Also, I think it confuses people when we talk about the relative emphasis or “importance” of RTCs. By “importance” we mean statistical power. However the lay public imagine we’re talking politics.
In mathematics you’re not an assh0le when you say, “my answer is correct and yours is wrong.” In politics, you are.
Anyone who understands Bayes’ theorem can see that the SBM guys are right and their opponents are simply wrong.
So why the arguing? Because two decades of alt med propaganda have normalized woolly headed thinking among the faculty of our medical schools. That fact is so incredible and so difficult to accept that, well, it will take time for people to get their heads around it.
@Dr Benway
You’re right. I should have called it a decision factor, not a value. Calling it a value confuses it with personal or public values. EBM assigns a factor of .01 to basic science results.
I think EBM tries to oversimplify (as I have done), and assign a factor in the decision a number based on methodology. In the old medical decision making field, the assigned numbers would vary based on the strength of evidence from different methodologies, as well as individual patient factors. That is why it is difficult to simplify the SBM approach to the same degree as EBM. It is not only the Bayes vs frequentist approach, it is the complexity of the decision process in SBM versus the simplicity in EBM.
I think the SBM vs. EBM dispute is mostly a matter of resolution.
EBM is mode of thinking suited for hordes while SBM strives to direct individual clinical decisions.
Mr. Simon (which I understand is a statistician) see’s clear facts as the gold standard of research. The multiple variables that make up an individual patient are for him “white noise” that needs to be minimized (most effectively by PRT). And the gathering of facts to form a scientific theory ,greater then its sum of parts, introduces an implicit uncertainty which lowers the statistical certainty (giving basic science its dismal location in evidence hierarchies).
Physicians like Dr. Novella on the other hand, are focused on individual clinical decision making. They are interested in a mode of thinking that will help in real-life diagnosis and treatment of patients and research that will advance their decision making.
No one (rational) is disputing the importance of EBM, best exemplified by the thalidomide example. A drug given arbitrarily to Multiple Myeloma patients without any scientific rational and found to be effective. In fact most of our medications were discovered more by luck than reason.
Importantly though, there was no science to indicate the contrary to the use of thalidomide in MM. Likewise, no one would try to actually test the parachute hypothesis by jumping of planes without them, because we have a basic scientific understanding of gravity (I think researching homeopathy is silly).
SBM basically sais that in the context of an individual patient EBM must be taken into account but moderated by our knowledge of physiology and science, this moderation is vital since patient uniqueness erased by PRT is crucial in a horde of one.
Just like how so many treatments was approved or rejected by looking at meta-analysis and systematic reviews though an EBM approach, why can’t it happen to homeopathy and acupuncture too? Is it the failure of EBM or it the insistence of these CAM groups to intentionally distort the evidence.
And when you talk about plausibility and is all about the treatment not violating the established science, the problem is that we don’t know if the explanation used is wrong -the observations might be true.
Your language is confusing me a little. Can I replace “explanation” with “hypothesis” and illustrate with an example?
Hypothesis: “homeopathy works”
Observation: an RTC showing a positive effect for homeopathy against placebo with a p = .05.
Yes, the observation is correct –meaning no fraud– in my example. But we *do* know the hypothesis is wrong, because of physics.
“Because two decades of alt med propaganda have normalized woolly headed thinking among the faculty of our medical schools.”
It’s a woolly mammoth.
Uh, no. Prior probability. That’s it. Don’t muddy the waters.
[i]Your language is confusing me a little. Can I replace “explanation” with “hypothesis” and illustrate with an example?
Hypothesis: “homeopathy works”
Observation: an RTC showing a positive effect for homeopathy against placebo with a p = .05.
Yes, the observation is correct –meaning no fraud– in my example. But we *do* know the hypothesis is wrong, because of physics.[/i]
No.
Explanation means the science used to come to the “hypothesis”. For example, acupuncture might be working with the release of endorphins which is plausible, but the CAM experts would say it’s working via meridians and energy flow which is not plausible. So the date might be true, but the causal mechanisms might be wrong here.
For the sake of those who haven’t read closely, JMB doesn’t mean literally “.01.” That’s just a made up number. The point is, EBM’s pyramid of evidential merit had “basic science” down low.
There’s a bigger problem. “Basic science” as a term is a lie. It has nothing to do with Nature. It’s merely a social construct humans find convenient.
Imagine a parallel universe in which the NEJM publishes about three physics or chemistry papers each issue. Now do your systematic review of homeopathy.
OK, but “causal mechanism” is where the equivocators hang out. You must anticipate them else you’ll have Dianetics rather than psychiatry at your hospitals.
Consider these two syntactically parallel sentences:
1. We don’t know how homeopathy works; we just observe that it does.
2. We don’t know how Prozac works; we just observe that it does.
There is a gross epistemic difference between 1 and 2. If it isn’t immediately obvious to you, then I have a bridge to total freedom you might be interested in purchasing.
When you say, “x works” you must define what you mean by “x” in concrete terms your grandmother can understand. If you can’t do that, you are simply saying “something works.” That and a cup of coffee will get you a cup of coffee.
“For example, acupuncture might be working with the release of endorphins which is plausible, but the CAM experts would say it’s working via meridians and energy flow which is not plausible. So the date might be true, but the causal mechanisms might be wrong here.”
But endorfin is released regardless where you prickle the patient. Plausibility question should be applied to the claim of acupuncture, i.e. that there are specific points that are active. Prickling the patient does not equal to acupuncture.
Exactly.
Whenever you hear, “x works,” be sure to find out precisely what is meant by “x.” Sometimes it means, “mystery sandwich.”
DO NOT WANT MYSTERY SANDWICH!!!! BEWARES!!!!!
The goal of medicine is not to formulate correct hypotheses, but to administer treatments to patients so that the health problems that patients have get “better”.
The patient getting “better” has several components, including the patient feeling better and also the patient experiencing an improved state of health via objective outside criteria. If all that was necessary was the patient’s subjective feelings, then administering a combination of cocaine, heroin and amphetamine would be the “best” treatment because no matter what the patient had it would make them feel really good.
Treatments that only influence subjective symptoms must be looked at extremely carefully to ensure that they are not simply acting the way that a combination of cocaine, heroin and amphetamine would act. One way of doing that is with clinical trials that look at objective endpoints. Another way is to understand the underlying physiology and how the treatment affects the underlying physiology.
A clinical trial that only looks at subjective endpoints (the patient feels better) is analogous to a trial of cocaine, heroin and amphetamine. If such a trial was made short enough, all patients would do very well. Fortunately we know some of the physiology behind administering a combination of cocaine, heroin and amphetamine and SBM would not consider such a trial in the first place. EBM can only look at what trials have been published, so if a supplier “games” the medical literature, bad treatments can have positive published clinical trials.
That is what CAM is doing now, “gaming” the medical literature and the clinical trial procedures to get their trials with only subjective endpoints considered.
@Dr Benway
You’re right again. 0.01 is just a fuzzy logic value (value in the sense of fuzzy logic theory) estimated from some of the published Cochrane articles that seem to lack input from other scientific disciplines (or for that matter, diverse clinical scientists). That’s a statement of my opinion. The formula was intended to demonstrate that SBM can increase the weight of basic science but avoid the trap of recommending treatments that consistently fail RCTs in spite of the body of evidence from more basic science. Although the formalization of EBM does not include such a formula, the relative weights can be inferred by the hierarchy of evidence. Fuzzy logic values can be inferred from the accepted papers published on the Cochrane sites (although those fuzzy logic values may not be what Sackett intended). The result of such a fuzzy logic assessment is not that we can quantitatively compare SBM and EBM, but that we can increase the importance of basic science evidence without completely devaluing consistent evidence from RCTs.
However I’m sure I bored most reader’s with the more rigorous description (even more than were bored with the less rigorous description).
**********************
# enosh askenasy
I think the editors on this site are also saying that the Bayesian approach should also be applied to the assessment of published research, and to decisions of allocation of public resources. It will be interesting if Ioannidis association with the Cochrane Collaboration will have such an impact.
My personal bias is that Sackett took some elements of medical decision making theory (that part evaluating how medical experts assessed published research) and dropped the rest (how research results can be translated into medical decisions). The Bayes approach had a big impact on the field of medical decision making. SBM would restore many of the Bayes implications of how medical research should be assessed, how resources should be allocated for research (and what constitutes clinical equipoise), and how we make decisions in medical practice based on assessment of research.
I also think that the trend toward incorporation of individual genomes into medical decisions will force some changes into the traditional hierarchy of evidence. RCTs will be necessary to prove a causal link. But individual decisions will require a Bayes approach because of the amount of information will will have to predict an individual’s physiologic response to intervention. Our risk vs benefits calculations will in many cases be based on individual factors, rather than matching limited individual characteristics to the selection criteria in randomized clinical trials.
*****************************
This is somewhat off subject, but is related to these comments. The recent USPSTF used fuzzy logic values (I am not saying their reasoning was fuzzy in a derogatory sense, but shared characteristics with fuzzy set theory). Although the calculated risk vs benefit estimate was 1 in 1904 lives saved in the age group 40 to 50 for screening mammography, they did not recommend screening because of poorly quantified risks of overdiagnosis and overtreatment. They were also recommending a fuzzy logic based decision (again a mathematical model, not a derogatory term) that women (aged 40 to 50) consult with their doctors for individual determination of risk factors that would play into the decision. My primary objection (not my only objection) was that there was enough data from the RCTs that the frequentist estimates of the parameters was sufficient to indicate that those fuzzy estimates had already been quantified, and included in the estimate of risk vs benefit of 1 in 1904 lives saved. The frequentist approach can give us reliable estimates that we can incorporate in a Bayes analysis. When reliable probabilities are available, we should use those probabilities rather than fuzzy logic estimates.
Dr. Benway,
Didn’t mean to muddy the water, I was trying to make an observation, obviously badly phrased.
What I basically meant was that SBM is in tune with the way physicians think. Prior Probability is the tool of our trade when diagnosing, it stands to reason that we will see it as a requirement for research fund appropriation. To a statistician’s way of thinking this is a foreign concept. You cannot numerically quantify prior probability, you cant build an algorithm around it (in my limited understanding). Its basically akin to “common sense”.
In the era before EBM the physician was at the center of medicine- research and clinical decisions. EBM’s importance is in its standardization of medicine trying to limit human error, In the process an unintentional “blind spot” as you called it developed.
Simply put, if physicians don’t know best than the nature of medicine is up for debate. If the field of battle is randomized trials we can study everything and anything (Alien crop circles for pancreatic cancer). If medicine is a statistical entity then hard facts are a priori, and it is in the nature of statistics to negate prior assumptions in the favor of concrete results.
JMB
I think the discussion about personalized medicine in the era of genetics is fascinating. But as you said, it is outside the scope of this discussion. Perhaps someone at SBM will “pick up the glove” and write about it.
The trouble not being addressed sufficiently (for me) by advocates here is how to obtain the prior. Call it plausibility or what-you-will since the words don’t matter, people will not agree on it. No mechanism to obtain it is proposed. No procedure to elicit it is proposed.
Trying to summarize what the bayes factor is may be what EBM is de facto doing. You thereby avoid fighting about the prior, which is supplied by the individual, and will vary. You can conclude the evidence (bayes factor) is small, and may end up writing it needs more study, but I read some of those sentences as “if you want anyone to believe this crap you’ll have to come up with more evidence, cause right now it sucks”, and not as statements that “the public should fund more studies of this”. I agree the authors of such things should permit themselves to be a bit more outspoken.
I agree basic science is part of the evidence, but it is very hard to quantify how it affects the bayes factor obtained from, say, clinical trial data. We are very poor at making those calculations. I can say something about why, but it starts to be a long story having to do with how many possible models for mechanisms to explain a thing there are, and that specifying that prior would take infinite time, which would slow down science a little.
enosh: “To a statistician’s way of thinking this is a foreign concept. You cannot numerically quantify prior probability”. Perhaps I don’t follow, but for a bayesian, that’s exactly what you do when it’s you making a decision. (Imagine we bet on the outcome of tossing a coin repeatedly or even just once – I guarantee I will be using an exact prior, and it’ll happen to have a beta distribution.) If on the other hand you are trying to convince the public, you often say something about how different the posteriors would be depending on differences in the priors, since you admit people will not agree on it. I wish some people here would admit people don’t agree about priors, and admit that this is a problem. Instead, what you will get back is anecdotal examples, as when the prior is “essentially zero” for the writer at least – but it’s not empirically determined to be true for any group of people, and no mechanism to determine it is proposed.
I thought that if you have enough research, that being able to calculate an original prior isn’t necessary.
Say I arbitrarily start with the assumption that the prior probability of homeopathy being able to cure pancreatic cancer is 50%. This is before I have looked at any research at all. Then I take that 50% and multiply it by the probability of the existence of water memory, and multiply again by the probability that water memory can be transferred to dried sugar granules, and again by the probability of a molecule of active ingredient being present in the remedy, then by the probability of magic being a real force in medicine, then finally by the probability that your RCT of homeopathy and pancreatic cancer is showing a positive result.
Or say I start with the assumption that the prior probability of homeopathy being able to treat anything at all is 50%, then multiply by the probability of all existing RCTs of homeopathy showing a positive effect.
(My appalling lack of basic statisical understanding is showing. I am sure someone will kindly figure out what I’m trying to get at and restate it as something that makes sense. Which the above does not, I realize.)
But anyway, I thought that you didn’t need to know the original prior; 50% is good enough because with homeopathy by the time you’ve reviewed the relevant research you’re going to end up with something fantastically miniscule anyway. And for something like “probability of water memory being transferrable to dry sugar granules,” since there is no supporting evidence at all you can arbitrarily (and exceedingly generously) set the probability at 0.001%. You will still end up with a fantastically small probability that any apparent positive result of an RCT reflects activity of homeopathic remedies.
(… is there any sense to extract here? Or am I just missing the point?)
rork
I stand corrected.
In many instances relating to CAM a bayesian statistician will find it takes an infinite time to calculate the prior probability derived from basic science, a practicing md such as Dr. Benway will do it in 3 seconds without using a calculator.
Everyone accepts that in medical decision making, gray areas exist where reasonable people may disagree. There’s no point in trying to make that problem go away.
But the fact that gray areas exist should not distract us from a more pressing problem: medical batsh1t.
With respect to batsh1t, you certainly CAN numerically quantify prior probability. There are several ways to do this. One I illustrated someplace around here recently: look at the published evidence that contradicts the hypothesis in question and calculate the statistical power required to overturn all that.
@rork
Sometimes I think EBM is a design by epidemiologists frustrated in trying to work with clinical scientists. For reasonable priors for medical hypotheses being researched, clinical scientists can be polled. One requirement is that the clinical scientist must be well versed in the field. A second requirement is that several qualified clinical scientists must be polled to obtain a sort of consensus. Statisticians may not like the imprecision of the estimates, but even with the imprecision, the quality of the resulting decisions can improve. It only takes a ball park estimate of .0001 with confidence intervals of .01 to .000001 to show that any positive benefit observed from administering a homeopathic remedy must be a placebo effect. In spite of the imprecision, that is still a better conclusion than reached by EBM.
JMB, I’m not sure I like the polling argument. I think we can do better than that.
Often we can use a little logic to reason out the implications of some hypothesis, and often we do have evidence pro or con those implications.
Intersubjective agreement should be a last resort.
‘I thought that if you have enough research, that being able to calculate an original prior isn’t necessary. ”
The problem is that you want to actually avoid doing that research when the prior is close to zero. With limited time, money, and manpower, you have to make choices. That sort of makes the priors essential.
@Dr Benway
I would agree with that. I was just trying to anticipate the counter argument that we would be picking priors based on our bias of what the expected outcomes should be. Based on my experience in computer aided diagnosis, if you qualify experts by measuring their success in similar decisions, you can arrive at a close approximation of the prior probabilities by intersubjective agreement that would be measured if you had the resources to conduct the formal observation. The testing of the experts is a slight modification of the intersubjective agreement process that improves its outcome. A lot of people can be in a position that would qualify them as experts, but measured performance on certain types of decisions would be a better qualification of expert status. The qualification process can counter the argument of bias. If Dr Atwood can successfully predict the outcomes of controlled clinical trials of homeopathy as hovering around the result of placebo effect, then it is hard to dismiss Dr Atwood’s estimated priors as bias.
Priors can be estimated by a few methods. They don’t have to be that exact to be useful. Low priors that exclude the possibility that we are observing anything other than a placebo effect don’t have to be exact. They can definitely fall into the grey area of precision. The priors (and posteriors) for which we need more exact estimates are those in which we are proposing intervention in the general public, for example the screening interventions. Small errors in precision can end up affecting large numbers of people (1% of 22 million is 220,000).
It is interesting that the government tried the qualification process and intersubjective agreement in predicting terrorist attacks. They used a sophisticated method based in game theory of identifying which analysts would have the best success in predicting attacks. Unfortunately, it got smeared in the public press as a ruthless gambling office pool in which people won money when they correctly predicted a terrorist attack. The public failed to understand how sophisticated a system it was for qualifying the analysts as experts, and thereby improving the precision of estimated priors (it was an intersubjective scheme). I guess you could say that the success of bookies is a measure of success of intersubjective agreement.
Yes, I understand signal averaging as a way to reduce noise. It’s a good idea. But a last resort. The vitaministas have become very wealthy and politically connected. They have figured out how to stack panels of experts in their favor.
Anyway, why do all that when we can simply say, “If homeopathy is true, the Standard Model is wrong. The evidence in support of the Standard Model is 19 billion papers with p<.0001. Therefore, any RTC study of homeopathy will require x subjects to achieve sufficient statistical power to overturn all of that, with x being a number so huge your head would explode if you saw it."
It’s more than signal averaging to reduce noise, it is a way of identifying which of those we call experts can give us fairly accurate estimates of prior probabilities. So it doesn’t average all estimates, just estimates of those with a proven track record. Furthermore, since the measure is based on prospective (or blinded to results) estimates of prior probability, then the selection process counters the criticism of bias in the panel (and there are statistical measures of bias that can be derived from the selection process). So Kimball Atwood should apply, but Dana Ullman needn’t bother. But if both were candidates, we could be comfortable that the selection process would reliably select the true expert. It is also a way for NCCAM in the NIH to dig itself out from the burial by woo.
Wouldn’t it be interesting if the editors of SBM were given the opportunity to predict the results of government funded experiments in CAM, and by weight of success of predictions, could eventually overrule the current panel used by NCCAM to grant funding for studies. That would certainly be in the interest of taxpayers. Or you could set up a system like the CIA, and allow anyone to bet real money on the outcome of proposed studies. Then the greater your winnings, the more your input would be weighted for granting funding of research. I’d rather bet on NCCAM results than the stock market (who wouldn’t?).
JBM,
Prior to becoming director of NCCAM, Dr. Josephine Briggs had a distinguished career as a genuine scientist with published work in nephrology.
My point is, the pro-woo crowd is very good at finding compliant experts. They’re just a lot smarter than we are politically (srsly, how many here give money to politicians regularly?) And, unlike bigPharma, they’re not under the watchful eye of a broader community of academic physicians.
The “panel of experts” idea is no guarantee against the rise of Dr. Lysenko. So, if we have a more objective means of separating sense from nonsense, we should use that first.
Sorry for not wading through the prior comments, but has anyone mentioned the danger of allowing prejudices and instinctive beliefs to affect our understanding as to what is plausible?
Evidence based medicine certainly has its flaws, but that forces us to acknowledge just how limited our current understanding of many medical matters still are. I worry that the presumption that we are capable of determining what is plausible and what is not will lead to a legitimisation of those instinctive beliefs to be found amongst amongst the medical community: those we all have, those especially common amongst those drawn to work in medicine, and those inculcated by the experience of medical practice.
If there is evidence showing that homeopathy can work as well as other, more plausible treatments in RCTs, then that’s good reason to look again at the plausibility of these more conventional treatments imo. RCT’s themselves have limitations, especially when it comes to subjective ailments, but this is a problem for plausible and implausible treatments alike, and some of the talk about science-based-medicine sounds worryingly like trying to put a finger of the scale because of prior expectations. Surely a poor way to do science.
I just stumbled upon this quote yesterday: “Science is the organized skepticism in the reliability of expert opinion.” – Richard Feynman.”
And if there isn’t?
“And if there isn’t?”
Then plausibility doesn’t need to come in to it. Evidence based medicine did fine without it.
Um, geo, how is it plausible for a chemical interaction (like curing influenza) with something diluted to one part to more parts than there are atoms in the universe?
Homeopathic Oscillococcinum 200C is one part duck bit to 10400 parts of solvent. There is only an estimated 1080 atoms in the known universe. So please explain how in this universe Oscillococcinum would have a plausibility of actually affecting a person’s biological reactions to influenza.
I’m not saying anything about the plausibility of homeopathy, which is almost certainly rubbish. I don’t know why you think I am.
Aren’t these your words:
Yes.
More completely: “If there is evidence showing that homeopathy can work as well as other, more plausible treatments in RCTs, then that’s good reason to look again at the plausibility of these more conventional treatments imo.”
So if similarly designed RCTs to those used as evidence for the efficacy of a conventional, plausible treatment could also be used to show homeopathy had a similarly beneficial effect, then we should look closely at the study design and examine whether the plausible treatment is really having any positive affect. If we were to have higher standards for the evidence needed in support of homeopathic interventions then that could allow plausible sounding treatments to be used despite not being any more effective than homeopathy.
By empahsising plausibility, rather than just evidence, science based medicine could make medicine less scientific and more easily influenced by the distortions of thought of its practitioners and researchers. (Also, as useless medicines go, at least homeopathy doesn’t have side-effects).
Okay, tell us what conventional treatments need to be checked for plausibility when compared to homeopathy. Because by bringing up the false dichotomy of “If there is evidence showing that homeopathy can work as well as other, more plausible treatments in RCTs” you are really confusing the issue.
geo, did you read the post? It discusses what is meant by “prior plausibility”. It does not mean what you think it means.
The problem with doing clinical trials on random things with no rational basis behind them (such as trials on homeopathy) is that RTCs are done on human beings, and human beings need to be treated respectfully and ethically and according to the Declaration of Helsinki. Otherwise it is a crime against humanity.
For a RTC to be done on humans, there must be “clinical equipoise”, that is the two legs of the trial must be clinically equivalent. One cannot be obviously clinically inferior. This is why a vaccinated/unvaccinated trial cannot be done. It is unethical to not vaccinate children against vaccine preventable diseases because those diseases are known to be much worse than the vaccine.
By what basis is the magic water produced by homeopathic principles thought to be clinically equivalent to any treatment with actual medicine?
@ chris: I was only speaking about abstract possibilities. Y’know – for philosophical fun. I was reading this a couple of days ago though: http://jme.bmj.com/content/early/2010/10/20/jme.2010.039313.abstract
@ daedalus2u:
So is the plausibility argument only used for deciding research funding then? I thought I’d read other articles here were it was being promoted as a way of evaluating the evidence from research? Maybe I am mistaken though – I had thought the article above said that the Cochrane review found positive RCTs for asthma and homeopathy.
I actually think that political considerations and the desires of patients should be given significant influence over decisions about research priorities, but that is a rather different discussion.
I am an engineer, I hate philosophy. Try something based on reality.
“I actually think that political considerations and the desires of patients should be given significant influence over decisions about research priorities…”
Yes, because politicians and patients know so much more about medicine than actual doctors do, so it’s only reasonable that they make the decisions about what gets research dollars.
Prior plausibility is based upon evidence. In homeopathy’s case, that evidence includes everything we know about physics, which is a lot.
I think the problem here is the Science based medicine is not well defined here. I think as someone suggested have a post which talks about what is SBM and what are the common criticisms and link on the About page.
Like,
What is SBM and EBM?
Why we need SBM?
What are the common criticisms of SBM?
What are the limitations of SBM?
Examples?
Have a flow chart explaining the steps to see which treatment is worth by the SBM definition.
I think these longs post without too many subheading makes it harder to figure out things,especially for newbies. I think all the good points get hidden in these longs posts.
Geo, you misunderstand again. SBM uses all available information all the time. SBM does not compartmentalize evidence and use it for only one thing and then discard it. SBM uses and continues to use all available information at all times and for all purposes. Data that doesn’t fit together raises questions that need to be answered until all the data does fit together. SBM resolves that by getting more data, doing more experiments, doing more analysis.
When SBM looks at the data on homeopathy, everything fits with the hypothesis that homeopathy is a placebo. Basic research and clinical trials point to the same understanding that homeopathy is a placebo and nothing more than a placebo. Clinical trials with selective publication are unreliable for determining if something is a placebo or not.
If homeopathy did “work”, then we live in an acausal universe where magic works. I understand that many people want magic to “work”, so they can wish for things and expect them to happen. That is the basis for The Secret, the power of prayer, and the motivation behind virtually all CAM. If we actually did live in an acausal universe where magic “worked”, then there would be hints of that in basic research. There are no such hints. Anecdotes of magic working are best explained by people wanting to believe that their particular magic works and so they misinterpret what happens as being caused by what ever magic they were trying to invoke.
The requirement for prior plausibility before clinical trials can be done on humans is not about not wasting research funding, it is about treating the humans in that clinical trial ethically. If politicians can waste taxpayers money on useless wars and tax cuts for the extremely wealthy, they can waste it on useless research projects too.
Hi daedalus2u:
It’s possible we’re misunderstanding each other, but I’m still not really clear as to how SBM improves upon EBM in it’s analysis of the available evidence. I understand why plausibility should be a guiding factor for deciding research funding (although this is another complicated topic, and I think that it is the desires of society and patients which should determine the aims of publicly funded research rather than the interests of scientists).
I don’t think there’s any realistic chance that homeopathy is anything other than placebo – but having a placebo which its proponents think is effective and want evidence that this is the case could be a more useful control than an unloved sugar pill. That respectable journals have published papers on the positive affects of alternative, treatments with seemingly implausible mechanism (often for conditions of which subjective assessments are required to measure impact of treatment) should mean that we are equally sceptical of plausible sounding treatments whose efficacy is supported by similar papers.
I’d prefer to talk in abstract about these philisophical approaches to medicine, but for the engineers amongst you: CBT is widely recommended for the treatment of CFS. In RCT’s patients who have received CBT tend to fill in questionnaires in a way which reports lower levels of disability (I’m ignoring all the problems with diagnosing a condition like CFS – the trouble with real world cases is that they’re always too complicated to talk about concisely). This paper followed up one of these trials showing CBT to be marginally effective: http://www.ncbi.nlm.nih.gov/pubmed/20047707 It used actometers to measure activity levels amongst patients, and found that there wasn’t a significant difference in activity levels between CBT and control group (although this was not mentioned in their initial paper). The writers of the piece, CBT proponents, concluded that CBT did effectively reduce fatigue levels (because of the questionnaires) but not through the mechanism of increasing activity. To me, it seems entirely possible that CBT was able to alter the way people answered a questionnaire, but had no real impact upon levels of ‘fatigue’ or disability.
I don’t really want to get in to all the complexities that surround CFS research, but I think that there is a danger that the fact CBT seems so very plausible to many within the medical community means that it is being over-promoted relative to the evidence we have in support of it’s efficacy. I think that there’s a real danger that ‘plausibility’ leaves room for prejudices and prior assumptions.
@Dr Benway
There is a difference between experts defined by credentials, and experts defined by results of tests of performance. By testing people who have research credentials for their ability to correctly predict the outcomes of experiments, promoting those who pass the initial test, and continued monitoring of their performance if they are promoted to the panel of experts, the process can be made more objective. It is an emphasis on outcomes as a test of whether an “expert” should continue to be paid for their decisions.
@geo
It is partly an optimization problem. If we value plausibility to highly, we become entrenched in current concepts and don’t move forward. If we value plausibility too lowly, we waste taxpayer dollars on bogus research, and allow too many preliminary results to become standard of care. The optimum classifier (plausible or implausible) will allow basic research into less plausible studies based on the availability of resources (taxpayers dollars) but erect barriers to the idea that each new RCT replaces the results of every previous RCT. It is not that hard to find a more optimum approach to the problem than currently suggested by EBM. The other side of the coin is that there are innumerable plausible hypothesis that were quite promising in lower level research that never panned out in RCTs. The end result is that the plausible hypothesis is discredtied as a viable treatment.
There are dangers in trying to submit placebo effect to RCTs. There is tremendous variability in observed effects, and the behavior of the healthcare provider has a big impact on the result, as well as the beliefs of individual patients. That will likely be a real money pit, as RCT after RCT demonstrates conflicting results. While certain placebo effects will record more powerful results (such as midline sternotomy and ligation of the internal mammary artery to alleviate debilitating chest pain), that does not mean the result of the RCT should be used in the decision to use which placebo. Ethics, economics, and the ability of the healthcare providers to keep a straight face, are better deciding factors.
I am happy to get into a discussion of CFS and CBT. I know a great deal about the physiology of both of them. The thinking that people used to assign CBT a reasonable prior plausibility stems from a misunderstanding of what prior plausibility means. CBT is a placebo (according to my definition: an effective treatment with effects not due to a pharmacological effect, surgical effect, or other physical effects but mediated through communication).
Placebos can be effective. Psychotherapy is quite effective. Psychotherapy is a placebo because the effects are not chemical or surgical. It turns out that placebos do tend to be effective for CFS because the underlying pathophysiology of CFS is low nitric oxide, and the primary therapeutic effect of a placebo is to raise nitric oxide by neurogenic means.
There are two usual mistakes that people make in considering CFS and CBT together. The first is that there is “nothing wrong” in CFS so CBT is a reasonable approach for a disorder that has no physiological basis. This is not correct. CFS has many well documented symptoms that are all completely consistent with low NO, and the main symptom, exertional fatigue is characteristic of insufficient mitochondria and more rapid ATP depletion in muscle has been demonstrated in CFS. Mitochondria biogenesis is regulated by NO, so low NO leads to insufficient mitochondria which will always cause CFS.
The second mistake is that if people with CFS get better with CBT, that means that there is no physiological basis for CFS. This is not correct.
The results of the study you linked to do not surprise me. Increased physical activity is not necessary to increase mitochondria levels. What is necessary is increased NO levels. Increase NO levels and there will be increased mitochondria biogenesis and a resolution of the CFS.
The authors make the mistake that most exercise physiologists make. It is not exercise that increases aerobic endurance, it is the compensatory mechanisms triggered by exercise that produce the increased mitochondria biogenesis which results in the increased aerobic endurance.
Expert opinion is useful when you lack facts, and the method you describe will weed out the more annoying Dunning-Krugers, so yay for that.
I promise to shut up if you concede that this method is awesome in comparison to dictionary fishing or consulting the stars, but less awesome than objective measures of variables of interest, and that we ought to think long and hard before we forgo the pursuit of objective measures in favor of the “this expert might be better than flipping a coin” test.
Oh rats, JMB, I forgot to address your point: the test to qualify the experts is objective. Yes, as you describe it, it is.
But the expert’s opinion itself is subjective.
I would agree that objective measures of prior probability are better than expert opinion, even when expert subjectivity is monitored continuously by outcomes. However, there are significant gaps in our knowledge base of objective measures of prior probability. In those decisions in which we lack hardcore data (such as RCT type data) qualified expert opinion (SBM) performs better than burying your head in the sand (EBM). So expert opinion qualified by testing and monitored for reliability just fills in the gaps, but does so better than appointing a panel of experts based on past credentials.
An expert’s subjective opinion can be measured objectively in a test. The difference in use of an expert opinion and a machine generated number, is that we must routinely calibrate the machine, but we must monitor the outcomes of the expert opinion.
In other words, I concede, but that is no reason for you to shut up. I enjoy reading your comments.
“What is necessary is increased NO levels.”
OH NO, NOT NO AGAIN!
The issue of prior plausibility is that we don’t know how to calculate it. But we do know that what ever “prior plausibility calculator” we use should never tell us that a true result is not plausible because every true result must be by definition plausible. Every true result is always consistent with every piece of data, basic research as well as RCTs. If we are using a “prior plausibility calculator” that tells us a true result is implausible, that is a fault of our “prior plausibility calculator” and not a fault of the true result, or how the true result is presented.
This is why the idea of prior plausibility should always be framed as a lack of prior implausibility. This is difficult for people to do because it is always easy to ignore data that doesn’t fit a hypothesis, or to stop looking for data if it is a hypothesis that you don’t like and then claim the proponents of the hypothesis have cherry-picked only the data that fits the hypothesis.
Your default has to be “I don’t know” if a particular hypothesis is plausible or not until you have spent sufficient time to understand the hypothesis and the data behind it and are able to articulate why it is implausible. Only data is acceptable for determining if a hypothesis is implausible because it is only data which must be consistent with reality.
Hi daedalus2u:
This is getting ever more complicated by peripheral issues… curse these real life examples.
I think that if CBT were providing useful and practical information that led to dramatic reductions in levels of disability, then this would be quite different to if CBT were simply altering the language with which patients chose to describe their level of disability and the way they chose to fill in questionnaires. You would class both of these as a placebo affect, but I think that the moral implications either has for the way in which CBT is promoted to patients would be rather different.
I’m afraid that I’ve no idea about NO levels, and tend to assume CFS is unlikely to have any one cause.
The mechanism by which CBT helps with CFS is neither of those. It can be real help. It is primarily a stress relief kind of effect. The individual has social interactions in a safe environment with a trusted individual and this reprograms the ANS to generate less superoxide and more NO. It is a relaxation response type of effect, a turning off of the “fight or flight” state.
The conditions that CBT helps are all made worse by stress. Stress is a low NO state and the adverse effects of stress are mediated by low NO. The “beneficial” effects of stress are too, but the beneficial effects of stress are only beneficial in the short term. The major “benefit” is the increasing of metabolic resources by turning off non-essential systems (systems that are non-essential in the very short term). In the long term there are no non-essential systems, so in the long term high stress is always bad.
I think most people promoting CBT don’t understand how it works. You don’t need to understand how it works for it to work. If you go into it expecting it to not work, it probably won’t. Since most explanations of CBT are probably wrong (I think that only the one I have given is correct), it is not irrational to think that the explanation the therapist is giving is bogus (because it is) and that the treatment won’t work (because the rational is bogus), which then becomes a self-fulfilling prophesy (because for any talk therapy to be effective there needs to be trust which turns off the “fight or flight” state).
I think the same is true of pretty much all placebos. I put the placebo of talk therapy in a different class than the placebos of CAM because (most) talk therapists are not fooling themselves and their clients. The important part of talk therapy is the relationship and the trust between the clinician and patient. Usually that trust can’t happen if either the clinician or patient is lying.
Hi Daedalus2u:
“The mechanism by which CBT helps with CFS is neither of those… this reprograms the ANS to generate less superoxide and more NO. ”
This is part of the problem I have with the emphasis on plausibility and science, rather than evidence based medicine.
I worry that it leaves more room for people to think they know more than they do. Given the confused state of current CFS research I don’t think we can be so so sure of what is actually occurring, or by what mechanism.
While I understand that there are extreme cases, like homeopathy, where our understanding of reality is such that we can be justifiably confident, prior to RCTs, that homeopathy is not effective – but I think that it is hard to restrain that faith in our own knowledge to only the most extreme of examples. People seem drawn to believing they know more than they do, and to having too much confidence in their judgements about how to best treat those they have authority over. I value science because it can be such an effective system for restraining our instinctive beliefs and prejudices… but it struggles to work effectively when the evidence is as lacking and ambiguous as seems to be the case with CFS – and a number of other medical conditions.
Anyway, thanks for sharing your thoughts.
geo, I understand where you are coming from, but there isn’t another way to do it. Prior plausibility has to be tied back to data. If you can’t do that, then you are not calculating a “prior plausibility”. The more data the calculation is tied to, the better the estimate of plausibility is.
It is very easy for people to think they know more than they do, and to denigrate what other people know. At Mark Crislip’s other blog (rubor, dolor, calor, tumor, 11/23/10) he has a nice discussion of that. Incompetent people think they know more then they do and think that everyone else knows less, competent people know there is lots they don’t know but think that everyone else knows as much as they do.
There isn’t a short-cut. If you want to estimate a prior plausibility you have to really understand the science involved. If you don’t understand the science and can’t tie that science back to data, then you are just guessing. Sometimes people guess correctly (as in CBT and CFS) but sometimes they guess wrong, as in the many treatments that fail in clinical trials. Sometimes a treatment is effective, but not for the reasons originally suggested, I think the health effects of statins are not due to inhibition of cholesterol synthesis.
The strength of SBM is that it continues to apply all data to the problem. If there is a mismatch between basic science and clinical trials, SBM keeps plugging away at the science until they do match. EBM just says do more clinical trials.
@geo
I don’t think the editors of this site disagree with the ordering of the hierarchy of evidence promoted by EBM. They just disagree about the relative weight given to each level in the hierarchy by EBM. So the weight of the lowest level of evidence is increased slightly, and the weight of the highest levels of evidence are modified (and often reduced) by prior probability. The highest level of evidence that is clinically feasible, is still required for justification of use in the healthcare. Conclusions that we need further research after meta analysis of clinical research into scientifically implausible mechanisms are not justified. Well designed RCTs can give us incorrect answers for implausible interventions based on the arguments by Ioannidis.
It is the editors of this site that define what SBM is, not those of us that make frequent comments. daedulus promotes nitrous oxide as a scientific mechanism. It is an explanation for many poorly understood physiologic effects, often mediated by the autonomic nervous system (if I understand daedulus). That is a justification for research, not clinical use. I promote the old fashioned decision analysis, mathematical modeling, and Bayesian approach that came from computer aided diagnosis research. Computer aided diagnosis involved the translation of scientific mechanisms, clinical expertise, and evidence into medical decisions, with success measured by outcomes. To me, EBM is an oversimplification of what we learned in the past.
JBM, nitric oxide, not nitrous oxide. Nitric oxide is NO and is a signaling molecule in many hundreds of pathways. Nitrous oxide is N2O, a not-so-great inhalation anesthetic (the mechanism of which is still not understood).
I disagree that under SBM only treatments that have been put through clinical trials are suitable for clinical use. There was the famous example of parachutes. SBM would allow (require) parachute use even in the absence of clinical trials because the prior plausibility is so high.
EBM wants to treat physiology as a “black box”, where everything going on inside the black box is inscrutable. SBM wants to scrape off the covering of the black box, look inside and try to figure out what is really going on. SBM wants everything to be transparent, EBM assumes everything except RCTs are completely opaque and then uses blinders to not look at it.
SBM is more powerful because it draws on more information. But with greater power comes greater responsibility. SBM is easier to misuse because it does allow more information to be considered. If that information is cherry-picked or distorted, wrong prior plausibilities can be wrongly “calculated”. We shouldn’t not use more powerful techniques simply because they can be misused. EBM can be misused too.
There isn’t a formulaic response to complex questions with limited data. EBM tries to force a formulaic response by limiting the data to only that generated utilizing RTCs. SBM adds everything else that is known to the RTC data, so SBM has the capacity to get better answers. But with more data come more degrees of freedom if only subsets of the data are looked at (actually more data reduces the degrees of freedom by adding more constraints).
If you only have the capacity to look at 100 pieces of data, using only the most reliable 100 pieces of data will give you a better answer. That is the approach that EBM takes, truncate the data to only RCTs. SBM says, lets look at 1,000 or 10,000 pieces of data, lets look at all the data that could possibly be important. If you do look at all the data, then you will get a better answer. The problem is when do you stop looking at data. If you stop when the prior plausibility exceeds your cutoff, i.e. when it reaches 0.5000001%, then you have cherry-picked the data and are not doing SBM.
Clinical trials under EBM are only possible when physiology is treated as a black box, when the patients in the clinical trial are all “equivalent”. If the patients are not equivalent, for example if you have a complete genome sequence of each patient, EBM can’t treat a set of those patients as being equal because they are not equal. More information (the genome sequence of each patient in the trial) turns the EBM RCT into a collection of anecdotes (because each patient is now unique).
As scientists we know that more data is always better, but EBM can’t handle more data.
@daedulus thanks for the correction. NO not N2O.
I personally have participated in an autopsy of a low altitude ejection. That is a clinical observation, not a clinical trial. Clinical observation is common sense with a trained eye. Well documented scientific mechanisms still need some input from clinical research, whether clinical observation or trials. Clinical observations of the rate of injuries from old style parachutes gave incentive to improve parachutes. They did not need a randomized prospective clinical trial to determine the need for replacing the old parachutes.
I think you are right about the error of EBM treating underlying physiologic mechanisms as a black box. Errors in research occur not only from low prior probability, but from mistakes in experimental design failing to adequately isolate the part of the physiologic mechanism that we wish to test.
re “It is the editors of this site that define what SBM is, not those of us that make frequent comments”:
Wallace Sampson has posted here a few times on CFS, saying stuff like:
“To those who still must argue that CFS and related disorders are diseases and not somatiform illnesses, and to patients who deny their problems’ origins, allow me to drop the fomalities of physician behavior and to speak in language used here.
“You are wrong. Stop looking for information that fits your conceptions, and try to learn what is happening to you; how your life – the only one you may have – may slip away under a pall of unhappiness and contrariness, while disallowing entry to sources that may help you the most. There is hope if defenses relent, and none if they are maintained. ”
Just as I don’t think the evidence is there to yet support daedalus2u’s claims, I don’t think it is there for Sampson’s either – and Sampson’s also seem more likely to lead to patients being treated unfairly. Other contributor’s have not condemned Sampson’s writings – indeed, Harriet Hall defended his comments thus: “Dr. Sampson stuck his neck out and stated a strong opinion.” Surely a defence that could be used for all manner of quackery (Quackwatch definition: Quackery can be broadly defined as “anything involving overpromotion in the field of health” – pretty similar to sticking you neck out with a strong opinion imo).
Given our tendency to exaggerate our own levels of knowledge, and to have a misplaced faith in those we see as experts, I think it’s better to aim for an undue scepticism of our own abilities – we’re shooting through a cross wind.
I’m still not entirely clear on the differences between EBM and SBM – it seems like much can depend upon the preferences of those using the terms, and some people’s conceptions of EBM would be identical to some people’s understanding of SBM. I think I’ve explained why some conceptions of SBM concern me, and that was all I intended to do. Thanks.
geo, not to be too much of a noodge, but you have to actually do a pretty good look at the data that is there before you can say you don’t think there is enough.
I understand that. I’m just reading a Maes and Twisk paper on ‘oxidative and nitrosative stress pathways’ (and reading back through the publication history) – I don’t think the evidence is there. I’m happy to be shown I’m wrong (although maybe that discussion would be rather OT here).
Surely you don’t think the evidence is there for Sampson though – so you should be able to appreciate my point from that perspective.
There is a lot of wrong stuff in the NO literature, especially as it relates to CFS. I have writen a blog about NO and CFS.
http://www.chronicfatiguetreatments.com/wordpress/treatments/chronic-fatigue-syndrome-nitric-oxide/
Only the blog and comments under my name are mine, there rest of the site isn’t.
The Sampson quote, let it go dude. Holding onto hurt feelings for months on end isn’t good for you.
Here, watch this video of the cutest parrot ever. You’ll feel better.
In defense of Sampson (who I completely disagree with, CFS is an actual, real physiology-caused metabolic state), the only “treatment” is to raise NO levels and so “reset” physiology to a high NO state. There are no accepted pharmacological treatments that will do that, there are only placebos.
If you have CFS, treating as if it is a somatiform disorder and trying to put yourself in a state where you are no longer “fighting” it, will help raise your NO level and will help in recovery. Treating it as something to “fight” will make it worse. Fighting with people who say it is somatiform will make it worse too. CFS is one of the things that trying to “fight” it makes it worse. Dr Benway is right, holding on to hurt feelings will make it worse too.
My bacteria will do it, but there is nothing published on my bacteria that demonstrates that they will. I have a lot of theory, and a couple anecdotes (including a recovery from ARDS that the MDs involved called “miraculous” and had no explanation for (and wouldn’t listen to mine)).
Thanks for the concerns, and I don’t mean to be personally condemning in my reply, but I think it’s socially dangerous to talk about our responses in such a pragmatic way. It undermines our sense of moral imperative and our commitment to believe what is most reasonable rather than that which we think will be most useful for us individually. One of my concerns about SBM is that it could encourage a faith in it’s own self-righting mechanisms that is seen to legitimise a reduction in our own individual moral responsibilities to speak out against such quackery.
Anyway, I think I’ve made myself as clear as I can do on this one.
deadalus2u:
IMO, if you want to be taken seriously, you really need to stop making claims like this. Without robust clinical data, you don’t even know that raising NO levels will actually treat CFS, much less that your bacteria can do that.
You think they’ll do it, based (as you say) on a lot of theory and a some anecdotes. But surely you know how many how many treatments have failed in the clinic, despite being supported by lots of theory and anecdotes.
Claiming that your treatment method will work, and that it’s the only method that will work, is the hallmark of a woo-meister. If you don’t want to be taken for one, you should make an effort to distinguish what you know from what you hypothesize. It’s also good scientific practice. Reminding yourself that you don’t actually know that your idea is right helps you guard against your own bias and assumptions.
@geo
I don’t think you have read very many posts by the editor’s, and perhaps you should reread this post. The editors have previously reminded commentors that they have an obligation to speak out about quackery, even when those are soft targets (when some commentors have pointed out that the editors are shooting fish in a barrel.) Tolerance of the editors for opinions expressed in this forum relate to the ethics of an open forum, not the moral obligation to slam every quack statement. Tolerance of the editors for statements claiming to be scientific assertions by commentors may result in a challenge by the editors (depending on how busy they are).
Both EBM and SBM have faith in the self-righting mechanism of British empiricism. However, SBM recommends greater skepticism about results of RCTs (including skepticism based on the arguments of Ioannidis). I don’t think any of the editors would accept the opinion that SBM is less skeptical and more faith based than EBM. If anything, EBM puts blind faith in evidence obtained from randomized clinical trials with adequate controls and statistical power, even though in science skepticism is required of any evidence and experimental design.
@ JMB – I should have been clearer. Wallace Sampson is one of SBM’s contributors, not just a poster in the comments section.
I’m certainly not saying that those at SBM would be slow to criticise those they see as not being a part of the SBM in-crowd… but what about those who are?
The evidence supporting Wallace’s views does not seem terribly compelling, but he has escaped censure partly because he is working withing the social frame-work of SBM and because his views fit with the prior beliefs many have about CFS. The blindness of the faith in EBM is something that appeals to me, as I think it will limit the overall level of trust and faith that seems to expand were we are able to be selective over its purview. Everyone wants their guys trusted, while the claims of opponents are treated dismissively – restricting our ability to make that distinction seems a good way of encouraging greater levels of scepticism.
Geo: The evidence supporting Wallace’s views does not seem terribly compelling, but he has escaped censure partly because he is working withing the social frame-work of SBM and because his views fit with the prior beliefs many have about CFS.
I agree with much that you say. I may be the only other to point out loud to weaknesses in the chains of evidence for daedalus2′s wide-reaching hypotheses on NO physiology. Yet he does make very valuable contributions here.
There IS considerable evidence favouring Wally’s views on CFS, at least regarding many of the patients so labelled. More importantly, on present knowledge, his approach may offer the best chance for most CFS sufferers to get back to a normal life, whatever initially caused the illness.
Hoping that one day other theories will reach fruition, validate the sufferer’s persistence in disease mode, and yield some pill that will magically terminate the condition without effort may condemn many to unnecessary wastage of a life, even if others of a possibly polyglot population will eventually be helped by the making of a precise pathophysiological diagnosis. They will then assume a different diagnosis.
This is a field that I only follow somewhat peripherally, so I am quite prepared to be shot down, so long as those pointing to this or that of the numerous abnormalities supposedly shown in CFS patients, have an answer to “chicken or egg” questions i.e. is this a primary cause of the condition, or the result of chronic inactivity prompted by other causes?
“There IS considerable evidence favouring Wally’s views on CFS, at least regarding many of the patients so labelled. More importantly, on present knowledge, his approach may offer the best chance for most CFS sufferers to get back to a normal life, whatever initially caused the illness.”
I’d really be interested to see what evidence. Whenever I look in to CFS papers I’m amazed by how weak they are (those written from a primarily psychological or biological perspective). This is a recent one that really shows nothing once you’ve looked in to the questionnaires used and references cited: http://content.karger.com/ProdukteDB/produkte.asp?Aktion=ShowFulltext&ArtikelNr=319312&Ausgabe=254424&ProduktNr=223864#AC
A collection of such papers can be combined for a meta-analysis – and then you’ve got what could be considered ‘considerable evidence’, but it’s all built upon nothing.
Lets get the Wallace quote again, before we examine it on pragmatic grounds:
“To those who still must argue that CFS and related disorders are diseases and not somatiform illnesses, and to patients who deny their problems’ origins, allow me to drop the fomalities of physician behavior and to speak in language used here.
“You are wrong. Stop looking for information that fits your conceptions, and try to learn what is happening to you; how your life – the only one you may have – may slip away under a pall of unhappiness and contrariness, while disallowing entry to sources that may help you the most. There is hope if defenses relent, and none if they are maintained. ”
Even if it’s not true that CFS is a somatiform illness, this would be acceptable because such an approach might offer patients the best chance to get back to a normal life?
This is so far from the way I think medicine should be done that it’s difficult to explain exactly why I think you wrong. Clearly Wallace’s claims fall within Quackwatch’s definition of quackery – I think this is bad. To make claims to patients without having clear evidence that they are true serves to undermine the trust that a good doctor-patient relationship requires (never mind that, ignoring the pragmatic considerations, it’s just wrong). Many CFS patients no-longer trust mainstream medicine because of quackery like Wallace’s. Even if you don’t know much about science or medicine, you just need to be of normal intelligence and have the vaguest of ideas about the state of CFS research, and how it is diagnosed, to realise that anyone making the sort of broad claims Wallace does is not to be trusted. To have those claims presented as being Science Based Medicine surely cannot be a good thing.
I’m so baffled by how anyone could try to defend Wallace’s comments that I’m not entirely sure if I’ve countered your arguments or not. Are you saying that we should treat all patients with CFS as if they’re suffering from a somatiform disorder, even though this is unlikely to be true, because otherwise they’ll waste too much of their time wishing for an honest explanation of their illness?
This rather pointless paper examines the views of CFS patients and medical staff, and included the quote: “We do need to be able to trust the people who treat us”. I don’t think that’s an unreasonable desire. http://www.biomedcentral.com/content/pdf/1471-2296-11-89.pdf
Have you read much from patients who have been treated as if their medical condition were psycho-somatic, only for a non-psychological cause to later be found? From what I’ve seen, they don’t tend to be grateful for having been saved the trouble of getting caught up in a sickness role. Living with mental health problems is incredibly difficult. Being treated as if you are mentally ill when you are not seems to scar people rather badly too.
In SBM, censure is reserved for those who hold and advocate views when there is clear and compelling data to the contrary. When the data is not so clear (as in CFS and how to treat it), not so much
. Vigorous debate is to be encouraged so long as people can disagree without becoming disagreeable.
I actually agree with Dr Sampson that the best course of treatment for a patient with CFS to proceed on right now, is to treat it as if it is a somatiform disorder with stress reduction and moderate exercise. Right now there isn’t a better treatment available. CFS patients should not wait until my NO bacteria become available. Even when they are available, my bacteria will work best along with the same treatments as for somatiform disorders, stress relief and moderate exercise.
It is unfortunate that there is neuronal remodeling that often accompanies CFS that is the perpetuation of the idea (which at first is actually correct) that exercise will cause pain and so exercise should not be done. When there are insufficient mitochondria, this is a protective mind-set. Trying to use ATP that one does not have the capacity to produce can cause serious damage because muscles can be worked to death, until they necrose. Immune system activation induces feelings of fatigue so that muscles are not used when they do not have the ATP generation capacity to support their activity. Humans generate all kinds of rationalizations to justify why they feel they cannot exercise. The rationalizations may not be correct and may persist after the need for them has passed.
Dealing with formerly adaptive ways of coping that have become maladaptive is extremely difficult. I think virtually all mental health issues fall into this category. Virtually all of them are adaptive in the short term (sometimes very short term, minutes) but maladaptive in the long term. Getting the regulation of everything back on track can be difficult because of the different time constants involved, the limited handles that can be manipulated, and the hysteresis of physiology. Changing feelings and behaviors requires changing neuroanatomy and for some types of change that is pretty slow.
I think the wanting to wait until there is a “cure” is partly due to the neuronal remodeling that CFS produces. In “the wild”, if you had CFS and could wait, eventually it would resolve itself (because in “the wild” everyone had a biofilm of my bacteria). Waiting doesn’t spontaneously work now because the biofilm has been lost through bathing. But overdoing exercise is very easy for people with CFS to do. They need to be very careful and calibrate the amount of pain they can push themselves to endure without causing bad side effects (more muscle damage than can resolve overnight). This level is idiosyncratic.
The low NO hypothesis of CFS does predict that there will be increases in neuropsychiatric disorders among those with CFS compared with non-CFS controls (because it is low NO that also causes neuropsychiatric disorders (I am working on writing this up)). But CFS is not caused by any type of mental health problem, both CFS and the various mental health problems are caused by the same thing, low NO.
PM, in CFS, the mitochondria depletion precedes the exercise intolerance. In the study on CBT that geo linked to above, the exercise intolerance of CFS resolved with CBT without an increase in physical activity. It is not physical activity per se that results in increases in exercise tolerance, it is the compensatory pathways triggered by physical activity that produce the mitochondria biogenesis that results in the increased capacity for aerobic ATP production that is necessary for exercise tolerance. The level of activity that produces a certain level of mitochondria biogenesis depends on the basal NO level. NO triggers mitochondria biogenesis, higher basal NO means that less NO generation by physical activity is necessary to trigger mitochondria biogenesis.
“PM, in CFS, the mitochondria depletion precedes the exercise intolerance.”
And you know this, how?
Geo, Wally would probably not talk directly to any patient that way, either, but I was mainly reacting to your implication that there is no good evidence for Wally’s position. The mere fact that many CFS sufferers can recover through simple interventions indicates that at least this sample of that population has no enduring underlying abnormality of their immune systems, mitochondria, or whatever.
I think patients would, or should, be approached on an individual basis. One can accept that the symptoms are real, as they are merely groos exaggerations of symptoms that are extremely prevalent in the supposedly “normal” population.
@ daedalus2u: Sorry for side-stepping most of your reply (which I’m afraid I disagree with much of), but I really am more interested in the abstract philosophical side.
“In SBM, censure is reserved for those who hold and advocate views when there is clear and compelling data to the contrary. When the data is not so clear (as in CFS and how to treat it), not so much
.”
When did the evidence to the contrary become so clear and compelling that it was no longer acceptable to promote the notion that black people were more closely related to monkeys than white? What evidence was it that excluded this sort of scientific racism from Science Based Medicine?
I don’t think that quackery was ever a legitimate part of science, and I think that the weakness of the evidence used by it’s proponents combined with the moral and social setting in which they were operating meant that they should have been censured from the start.
People should be free to believe what they want. Certainly, researchers need to be free to develop hypotheses, gather evidence and test their predictions; but I think that to promote your beliefs as facts in a forum like SBM you should already have clear and compelling data.
@ pmoran… It seems like you’re defending a reasonable position, and trying to presume that this is what Wallace meant to say. If you look at what he actually wrote, then their meaning is quite clear.
This is one of the things that worries me. You wouldn’t be trying to so generously re-interpret Wallace’s words if you did not already think that he was one of the good guys. These sorts of cognitive biases are ever present, and I think that the systems of knowledge which work best are those that leave the least room for them.
Anyway, it seems like I’ve gone full circle, and I can end with the quote I used in my first post. How poetic:
“Science is the organized skepticism in the reliability of expert opinion.” – Richard Feynman.
PM, Because there is considerable research demonstrating it.
CFS often occurs acutely just after severe immune system activation, as in septic shock. Sepsis does cause mitochondrial depletion. Mitochondria are the only source of aerobic ATP. ATP depletion accompanies muscle fatigue (as measured in vivo by MRS). In CFS there is faster and more severe ATP depletion that takes longer to resolve than in non-CFS controls. There is measured mitochondrial depletion in CFS patients. Here is quite a nice paper on it.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2680051/
The fatigue of dilative cardiomyopathy and of COPD is similar to that of CFS in that it affects the mitochondria and muscle endurance and not muscle strength (so much). The fatigue of COPD and dilative cardiomyopathy is due to insufficient mitochondria in muscle and not due to insufficient O2 delivery.
In normal controls it is heart output that limits VO2max. In CFS, DCM, COPD it is not heart output that limits VO2max. I am pretty sure that the fatigue of CFS, DCM, and COPD are all pretty similar and variations of similar pathways that reduce mitochondrial biogenesis.
The anaerobic threshold is reduced in CFS, that is the transition to glycolysis occurs at a lower work output. The respiratory exchange ration (CO2 produced/O2 consumed) is higher in CFS patients (indicating greater oxidation of carbohydrate than fat) and increases faster with exercise than in normal controls. The pH in CFS skeletal muscle drops faster with exercise (indicating more glycolysis sooner) then in controls.
geo, there is very robust data that says all humans are equally distant from all non-human ancestors. That is always true of all well-speciated species. Humans are well-speciated. All that means is that all humans share common ancestors after human speciation. If humans didn’t share common human ancestors after human speciation then humans would be multiple species instead of just one species.
That data has been well known for a long time, at least since the data put human speciation in Africa. There was never any data that said otherwise. There still are people who don’t understand the data and tried to spin it to show that Africans are more closely related to non-humans than non-Africans are, but that is simply false and easy to demonstrate that it is false.
@geo
You are more familiar with previous posts than I. Wallace Sampson’s last post occurred before I started following SBM. I did read his last post on CFS, and I think Mark Crislip’s comment was a better expert opinion. So I thank you for pointing that out.
I do like the quote from Richard Feynman, but I also think that qualifying expert opinion by tests of predicted outcomes by the “expert” does constitute one form of organized skepticism. In Sampson’s last post, we could guess that he is predicting that an infectious etiology will not be found. Mark Crislip makes a different prediction,
So by 2015, one will be a qualified expert based on the test of predicting outcomes, the other will be discredited in spite of previous credentials.
PM, it is well known that there can be substantial mitochondria damage during sepsis and that this damage is repaired over time (to some extent). Experiments in rats have shown mitochondrial DNA deletion during sepsis, and that this deletion resolves over time.
That damage is resolved by deleting the damaged mitochondria via autophagy and generating new mitochondria via mitochondria biogenesis. The regulation of mitochondria biogenesis is critical to the resolution of the damage done to mitochondria during sepsis. If there is something that impairs the regulation of mitochondria biogenesis (which happens to be regulated via NO), then the resolution of the mitochondrial damage that occurs during sepsis will be delayed. If it is delayed long enough, the patient experiences CFS. The CFS will resolve when their mitochondria status is restored to normal which can happen spontaneously.
Geo, I think censure is not the right term. Censure is a severe sanctioning for pretty egregious violations of group norms. In no way do I think that Dr Sampson should have been censured for what he said, even though I disagree with the factual basis of his reasoning. Treating people with CFS can be extremely challenging. I am glad that IANAD and so I don’t have to do it. I think that Dr Sampson’s anger was aimed at the quacks who are exploiting vulnerable people with CFS and not so much at the people with CFS who are allowing themselves to be exploited.
There really isn’t a better treatment modality for CFS right now. There is no shortage of quacks offering the hope of a better treatment but they are all false hopes. I can say this because I really do understand the physiology behind CFS. I appreciate that my say-so is not a sufficient response, but until I can get the resources to do clinical trials, that is all I have. I am sorry about it too.
Doctors are people too, and they can get very frustrated when needy people make bad choices and do things that are perceived to be self-destructive. There has been a lot of talk about how to deal with patients who don’t exercise, won’t stop smoking and who don’t eat a healthy diet. Is it “enabling” to give them cholesterol lowering drugs or is it good medicine? How much coercion is appropriate, helpful or counterproductive is very much a part of the idiosyncratic doctor-patient dynamic. Because that dynamic is always unique, there can’t be randomized trials on it because each case is only an anecdote. You can’t do EBM if all you have are anecdotes.
Daedalus2: PM, it is well known that there can be substantial mitochondria damage during sepsis and that this damage is repaired over time (to some extent). Experiments in rats have shown mitochondrial DNA deletion during sepsis, and that this deletion resolves over time. ‘
Not surprising and not obviously relevant to the usual case of CFS.
You referred me to this –
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2680051/
I had found this abstract myself. Are these findings well-replicated? It also does not answer my question as to whether the observations are contributing to the CFS state or secondary to muscle disuse. Perhaps simple exercise can over time reverse these observations.
I also wondered how seriously to take this work — the authors referred to the use of “supplements and detoxification” as treatment for CFS.
Do you have anything else that support your claim that “PM, in CFS, the mitochondria depletion precedes the exercise intolerance.” ?
I have no evidence that it doesn’t, and I wonder how that could ever be established in the usual case. My concern is
that you might start dotting your I’s and crossing your t’s
before being so dogmatic about certain conjectures. This is meant as friendly advice.
Sepsis does cause mitochondrial damage. This has been replicated many times in animals, here is a result in humans.
http://www.ncbi.nlm.nih.gov/pubmed/12133657/
Human and animal mitochondria are virtually identical, so animal results are directly translatable to humans. This is an excellent paper and shows both the increase in ATP due to the high NO from iNOS during sepsis (supported by glycolysis because mitochondria are inhibited by the high NO and high ATP) and the decline in mitochondria function following the fall in ATP. It is the fall in ATP when glycolysis cannot maintain the ATP level above where mitochondria stay “off” that causes mitochondria to “turn on” and in the high NO environment of sepsis, that kills them by inhibiting MnSOD.
CFS often follows sepsis. The “fatigue” following sepsis is not due to disuse during the few days of sepsis, it is due to the mitochondrial damage that occurs during sepsis. The “fatigue” following sepsis is not apparent until the patient tries to exert themselves and use more ATP than is needed “at rest”.
Infection is not the only precipitating factor, trauma, allergic reaction, surgery, and no precipitating event have also been reported.
http://www.ncbi.nlm.nih.gov/pubmed/9201648
You are correct, in the usual case it cannot be determined if the mitochondria depletion preceded the exercise intolerance because exercise intolerance is the first symptom to be noticed and so muscle biopsies cannot be done in the pre-symptomatic patient to determine mitochondria status in the pre-exercise intolerance state. But when the noticing of the symptoms of CFS is immediately preceded by a febrile illness, and a not uncommon characteristic of febrile illnesses is mitochondrial depletion, an inference of mitochondrial depletion is rather straightforward. To me, it would be perverse to postulate that exercise intolerance preceded mitochondrial depletion.
In some cases simple exercise over time can resolve symptoms of exercise intolerance. But not in all cases. Since CFS is defined to be cases where simple exercise over a specific time do not resolve symptoms of CFS, then those cases are not what I am talking about.
The paper doesn’t report anything that is unexpected, but it isn’t that good a paper and isn’t what I have based my formulation of CFS on. It measures ATP levels in neutrophils, which are not the right cells to use, muscle cells would be better, but muscle biopsies are a lot more invasive. Muscle ATP and neutrophil ATP do correlate. I didn’t look at it as carefully as I should have before linking to it. The abstract does talk about using supplements and reporting the results in a future paper. In looking at the author, she has just had her license to practice medicine restricted. Not a surprise because supplements won’t fix the mitochondria problem of CFS.
I have spent many years and many thousands of hours doing research on NO and NO physiology. That knowledge base cannot be compressed and conveyed in its entirety in comment-sized nuggets. CFS is not the topic of this thread, and hijacking the thread to go into the details of physiology of CFS isn’t something I want to do right now. I have an extensive write-up I did a few years ago that I could send you.
geo, this isn’t the first thread you’ve derailed for the sake of your grudge against Sampson.
Science based medicine is not a cult, a political party, or a special interest group. It doesn’t protect its reputation any more than mathematics protects its reputation. Everyone here could be arrested tomorrow for selling crack; wouldn’t change science based medicine.
There are gray areas in medicine where reasonable people may disagree. CFS is one of those gray areas, as a clear etiology has not been identified and the symptoms are non-specific.
Daedalus2: I have spent many years and many thousands of hours doing research on NO and NO physiology. That knowledge base cannot be compressed and conveyed in its entirety in comment-sized nuggets.
Nevertheless, this is how you will be challenged whenever you start making precise clinical claims.
What is the incidence of severe sepsis in patients with CFS, as usually defined? A significant period of fatigue after such an episode would not normally be so classified would it?
deadalus2u,
The original topic of this thread was how EBM (as currently practiced) undervalues basic science. In your case, I think it’s the opposite. You overvalue the basic science. Especially the science that fits your beliefs regarding NO.
No. Human and animal mitochondria are virtually identical, so animal results are likely to be directly translatable to humans. Humans are virtually identical to other animals in almost every aspect of their biochemistry. Yet drugs that work in animals fail to work in humans all the time!
I appreciate that you’ve devoted enormous time to studying the literature on NO. I acknoweldge that the available data may fully support your hypotheses on how NO and your bacteria might treat disease in humans. But until you have robust clinical data, you cannot rationally conclude that your ideas are right.
Those are interpretations of data. They are not facts. Fatigue following sepsis may be due to mitochondrial damage. It seems like a very reasonable hypothesis, but that’s all it is – a hypothesis.
Every time you make an unwarranted claim along the lines of “My bacteria will do it,” I become less inclined to believe any of your other claims. Such overconfidence ignores the long history of ideas that seemed to make perfect sense from the basic science, yet still didn’t work. If you’re willing to ignore that, why should I trust that you haven’t ignored other evidence that might contradict your claims?
EBM practitioners are wrong when they assume that clinical data always trumps basic science. But they are correct to insist that basic science can’t trump clinical data, and that basic science is very rarely (if ever) adequate in the absence of clinical data.
Maybe I’d say, basic science plausibility is a necessary screen but an insufficient measure of clinical plausibility.
Biochemistry and physiology result from genetic algorithms. Consequently, they’re impossible to debug in a systematic manner. Small changes in the genetic code have downstream effects no one can predict.
No qetzal, as Nietzsche said, “there are no facts, there are only interpretations”. There are only hypotheses. No quantity of clinical data can turn any treatment modality from one driven by a hypothesis into one driven by a fact.
That my positive statements about what my NO bacteria can do negatively affects your assessment of the plausibility that they can do what I say they can do is about your use of non-data driven reasoning methods. My positive statements either derive from facts and valid reasoning, in which case they add some positive probative value to the premise, or they are false delusions of fantasy in which case they add zero probative value. In no case can they add negative value. I appreciate that this is one of the most difficult aspects of SBM to apply, that any false statement has no effect on the Bayesian plausibility. There are infinitely many false statements that can be made. None of them have any effect on the truth value of true statements.
You think my claims are “unwarranted” because you do not have the knowledge base to evaluate them. If you did have the knowledge base to evaluate them, you would know that they are not unwarranted. I appreciate that you are unable to appreciate that. I am trying to increase the knowledge base of people who read SBM. Sorry if my tone of certainty is off-putting. There really is a very high plausibility, approaching virtual certainty that I am correct. I always admit the possibility that I am wrong. As Stephen Jay Gould said:
“In science, ‘fact’ can only mean ‘confirmed to such a degree that it would be perverse to withhold provisional assent.’ I suppose that apples might start to rise tomorrow, but the possibility does not merit equal time in physics classrooms.”
Provisional assent can be achieved via Bayesian analysis applied to a large body of knowledge. That is all that I am doing. That body of knowledge does not yet include RCTs, it does include some (unpublished) clinical vignettes, including a person who had ARDS, used my bacteria post-hospitalization, and in 2 months had gained 9 pounds, had gone hiking, had resumed his PhD program. His doctors called it a “miracle”, had never seen such a recovery, and had no explanation (and didn’t want to listen to mine). His recovery might have been a coincidence. It actually surprised me. I expected somewhat better improvement, I didn’t expect “miraculous” improvement (his doctor’s words).
Putting caveats in what I say that I don’t think are warranted would feel like I was being dishonest and disingenuous. I don’t go around using the caveat “unless gravity stops working” for the same reason. If you have any data that is in conflict with anything I have said, I would be glad to hear it, would look at it and very seriously consider it. If it is reliable data, I will incorporate it into my conceptualization of reality and if my prior statements about my bacteria no longer fit, I will no longer hold onto them.
Right now there is no data (that I am aware of) from basic science or clinical research that is incompatible with any statement I have made about my bacteria. There is a lot of data that is incompatible with other interpretations.
Dr Benway, for some interventions, clinical trials are not necessary and would be unethical to perform. The example of the use of parachutes has been given before.
You can’t use a “one size fits all” heuristic in extremely complicated systems no matter how much you want to when the system is much more complicated than your heuristic.
I think another way of stating the issue is that for evidence from clinical research, we must be careful in interpretation of observed effects. Classification of observed effects in clinical research must differentiate non-specific and placebo effects from the desired physiologic effects altering the disease process. For treatments that have no possible effect based on physics and chemistry, any evidence of effect observed in clinical research would be classified as a non-specific or placebo effect.
So the RCT is the most powerful scientific method we have, but the interpretation of results must account for non-specific effects and placebo effects. Treatments that are incompatible with current knowledge in physics and/or chemistry must be classified as placebo, until knowledge in physics and/or chemistry changes.
It is not just a matter of prior and posterior probabilities, it is also a matter of classifying the effect to differentiate physiologic effects from non-specific and placebo effects. Classification matters because the most predictable benefit in the translation of research results to general population use comes from physiologic effects on disease processes. Attempting to reproduce placebo effects from research in general population use has limited success. Non-specific effects generally do not translate into the population use.
The ultimate goal of science is to advance knowledge. The ultimate goal of EBM or SBM is to advance health in the general population. By using basic science as a classifier to differentiate physiologic effects from non-specific and placebo effects, SBM will be more successful in predicting the result of a healthcare intervention applied to general use, than will EBM.
@daedulus
Clinical research is still required of your bacteria. To compare your proposed bacteria use to use to the parachute analogy published in BMJ is incorrect. Even a parachute is dangerous if someone tries to drag a deployed parachute behind the while walking through downtown NYC traffic. Clinical research is necessary to define the parameters for indication for use (for the parachute, the indication for use is when you are falling a great distance), management of patients undergoing treatment (don’t deploy the parachute from a back pack when your back is facing the ground), and dosage requirements (you need a parachute of a certain size to be effective). Even water and oxygen are toxic at certain levels of intake.
deadalus2u,
Wrong. Your claims ARE unwarranted because you’re expressing virtual certainty about a clinically-untested intervention.
I don’t have the knowledge base to evaluate whether your hypotheses about NO are supported by the available data. That much is true. But I do have the knowlege base to know that basic biochemistry and animal studies are NEVER enough to support your level of certainty. That is even more true for a disease like CFS, which has no validated animal model.
You can play word games with the meaning of “facts” and “interpretations” all you like, but you would be better served by some honest examination of your statements and beliefs. Look how you’re using anecodotes to try to bolster your claims. You’re even throwing in the old “His doctors called it a ‘miracle’” bit! That’s the same stuff we hear from every other purveyor of woo. It’s baloney when they do it, and it’s baloney when you do it, too.
qetzal, so you don’t wear a seat belt because there are no RCTs of seat belt use? You don’t wear a helmet when you ride a bike because there are no RCTs of helmets and bike use? You don’t use a smoke detector because there are no RCTs of smoke detector use?
JMB, when was the RCT that showed it was dangerous to wear a parachute in NYC traffic? I must have missed that one, could you give me a link?
You do have a link don’t you, you are not merely asserting as fact the hypothesis that wearing a deployed parachute in NYC traffic is dangerous? I would put a pretty high prior plausibility on that hypothesis, but the way to assert it, with such confidence, I am sure you must have “data” to back it up. I am surprised that any IRB approved such a study. I would have thought it would be unethical. 
qetzal, no again. What you are talking about is methodolatry, worshiping the method of the clinical trial and damning everything for which there is an anecdote.
An anecdote is data of very limited statistical significance and limited probative value. The probative value of an anecdote is never negative. Taking a body of data that supports a hypothesis and then adding a positive anecdote to that data does not reduce the plausibility of that hypothesis. CAM modalities are wrong, not because they are supported by anecdotes, but because they are not supported by a body of data. Adding a positive anecdote is at worst like adding something completely extraneous, something that doesn’t follow, something that is unrelated to the question being asked. In that case is adds zero probative value. In no case does a positive anecdote take away from positive data.
I agree that there are many examples where basic research data is insufficient and that clinical trials are needed. Clinical trials are not the perfect source of information that the methodolatry that has grown up around them would indicate. All measurements, clinical trials included, can have type 1 and type 2 errors, false positives and false negatives. That is why you need lots of data, and why you need to add the data together using Bayesian analysis. That is really hard to do, especially with data that is complex.
Once you have done that Bayesian analysis with non-clinical data and that analysis shows that an intervention has a very high likelihood of helping (99%+) and a very small likelihood of not helping (1%-) and a negligible likelihood of harm (less then 0.00001%), how should one talk about it? A clinical trial doesn’t have the resolution to measure the difference between 99% and 98%. Any RTC that started to show a 99%+ treatment effect would be stopped early to give both legs the treatment because it would be unethical not to.
Suppose you did do a trial of wearing a parachute in NYC traffic. When would it be ethical to stop it? How about a trial of wearing a helmet while riding a bicycle? Those would be unethical from the start because there is no clinical equipoise between the two legs. One leg is clearly inferior based on prior plausibility.
Ethical considerations don’t allow you to do a clinical trial when one leg is clearly inferior.
daedalus
Interventions with that kind of risk vs benefit ratio usually become a general recommendation such as, don’t smoke, eat a balanced diet, and get exercise. With that kind of risk vs benefit ratio, I would say you have discovered vitamin NO.
I remember a genius organic chemist who was opposed to fluorine additive to drinking water, and chlorine treatment. In spite of the high prior probability. the risk vs benefit ratio expected from laboratory research was not reproduced in clinical observations, or population outcomes. There are significant limitations to our understanding of physiology that limits our translation of evidence from test tube to an intact animal, and from an animal to humans (or even one human to another).
JMB, you are exactly right. The “normal” state is with a biofilm of my bacteria on the external skin. It is like a “vitamin” that one absorbs through the skin from commensal bacteria, sort of like vitamins you can get from your gut bacteria.
The abnormal non-physiological state is to bathe every day and wash the biofilm off. There were no RCTs when people first started to do that, and the practice has become “grandfathered” and so ingrained that no one can even think to question it.
That is why I can be so confident there won’t be any adverse effects. It is like the “adverse” effects of a nutritious diet, sufficient sleep, and not smoking. Restore the physiological default condition, and health will improve because physiology is working at a more optimum operating point.
No, deadalus2u.
You don’t seem to be reading what I write. I’m not damning your basic science and anecdotal clinical evidence. I’m only saying that it ISN’T ENOUGH!
If there’s any methodolatry here, it’s from you. You are convinced that basic science is sufficient for you to be virtually certain (your words) that your proposed clinical interventions will be effective. That’s where you’re wrong. Basic science and anecdotal evidence is only enough to establish a reasonable hypothesis of clinical benefit.
And are you really trying to compare treatment of complex human diseases to smoke detectors or seat belts? If the former were as simple as the latter, you and I would have no dispute. But it’s not, as you well know.
You do NOT understand human physiology so well that you can predict with virtual certainty all the things you claim for NO. No one does.
Well I probably wasn’t clear. I was referring specifically to making predictions of human biochemistry or physiology on the basis of things we know in more basic sciences, such as physics or chemistry. In that context, we can reject hypotheses that outrage basic science but we cannot accept novel biochemical approaches without some empirical data.
Systems built upon random events aren’t predictable.
No qetzal, if the intervention I was using was non-physiologic, you would be absolutely correct.
I will go even farther and say that there may never be a non-physiologic method of changing basal NO levels. Probably never. Because the physiology is too complex and multi-factorial. NO/NOx physiology is too complex to control via the open-loop control that is the only mechanism available with standard pharmacological practice of oral, topical or injected drugs.
Maybe when nanobots are available and they can monitor physiology and deliver the right species of NO at the right time and in the right place and in the right amount, but that is many decades away, perhaps centuries, perhaps never.
Until then, my bacteria on the external skin will be the only method of bringing NO/NOx physiology closer to optimum. The only reason my method will work and others will fail is because my method is what physiology evolved to use. Until you restore the normal background and the normal physiology for regulating that background, NO/NOx physiology is not going to work right. For some things the misregulation won’t be that bad, but for some things it will be depending on the idiosyncratic physiology of the individual.
I do understand physiology well enough to know that it is self-regulating and so complex that artificial regulation won’t work until that artificial regulation can be as (or more) complex as physiology itself. I am not trying to apply complex regulation to physiology. I am simply restoring a normal part of physiology that modern practices have removed. That restoration allows physiology to better self-regulate. It is the better self-regulation that will produce the improved health effects that I am suggesting.
I don’t need to understand the details of physiology to know that removing non-physiological impediments to normal physiological regulation will improve self-regulation and improved self-regulation will lead to improved health. Maybe people won’t want improved health that improved NO/NOx will provide but that is a different issue.
Let me use another example, suppose there was a society where everyone was given a mask at birth and always breathed through this mask that restricted air flow. Suppose that people grew up always breathing through this restriction and never knew about breathing without a restriction.
Suppose in this society that there was a disease called hypoxia which resulted from insufficient O2 delivery to tissues. Suppose someone came along and claimed he/she could cure this hypoxia by removing the restrictive mask that everyone breathed through. Suppose this person had studied physiology and had come to realize that the reason everyone was hypoxic was because of the non-physiologic masks that people were breathing through, and that by removing the masks, the supply of O2 could be easily and physiologically increased.
Suppose there were nay sayers who said “what about O2 poisoning”? To which the reply was “there are physiological mechanisms for controlling O2 delivery, breathing without the mask is not a risk”.
Suppose now that this disease hypoxia was killing millions of people every year, and before it killed them it caused all kinds of bad symptoms, fatigue, dementia, heart disease, obesity, liver failure, depression, psychosis. Suppose there was pretty good evidence linking a lack of O2 to all of these diseases.
daedalus, natural selection does not favor optimum health.
The unit of selection is not an individual but a gene. Genes don’t get sick. They replicate, or not.
Dr Benway, “optimum” health is that health which maximizes the replication of the genes. Generation of “optimum” health is what the genes do to ensure their replication.
Many “disorders” are not disorders at all, they are “features”. Anaphylaxis is a feature, depression is a feature, Alzheimer’s is a feature, ischemic preconditioning is a feature.
They are all features that are adaptive in short term specific circumstances and maladaptive in the long term.
For the most part it is low NO that triggers the physiological subroutines that are adaptive in the short term and maladaptive in the long term. If NO does not go back up after the short term crisis has passed, then the “features” become maladaptive in the long term. The only way to fix it is to raise NO levels to turn off the short term adaptation that will become maladaptive in the long term.
Linus Pauling was a pretty smart guy. Smarter than most of us, I imagine. He knew a thing or two about biochemistry, too. And based on his very considerable intelligence and knowledge, he was convinced to a virtual certainty that megadoses of Vitamin C would have all kinds of therapeutics benefits. Treat cancers, prevent colds, etc.
But despite his virtual certainty, he was wrong. He made what is arguably the worst mistake a scientist can make. He stopped being sufficiently skeptical of his own ideas. He convinced himself that he knew so much, that he could predict with virtual certainty how his ‘orthomolecular medicine’ would work in people without any rigorous clinical testing.
There is no such thing as virtual certainty when it comes to predicting human response to a novel and untested physiologic intervention. But I guess you’re unable or unwilling to accept that. I won’t bother to try to argue it further with you.
qetzal, the heuristic you are using is pretty good for non-physiological interventions. For something non-physiologic, I completely agree that you need good clinical data in humans to have good confidence that the intervention will be helpful.
Linus Pauling was not talking about a physiological intervention. Supraphysiological doses of vitamin C are not a physiological intervention.
Nitric oxide from a biofilm of ammonia oxidizing bacteria produced from ammonia released by the skin is a physiological intervention.
Have there been any clinical trials demonstrating that the non-physiological intervention of bathing every day has no adverse effects? Is there any data that shows there are no adverse effects? I appreciate that there are lots of anecdotes, perhaps billions of anecdotes. As I am sure you are fond of saying, the plural of anecdote is not data.
Why do you choose the non-physiological state of the absence of a biofilm as your default condition? Humans have only existed in that state in the past few hundred years, since the advent of piped hot water for bathing, the same time frame as many of the modern diseases (heart disease, allergies, obesity, kidney failure, stroke) have become leading causes of death and when the average age of menarche dropped from ~17 to ~12. There are humans who do exist with a biofilm of these bacteria, people living in “the wild” where they never bathe. Such people have a very low incidence of heart disease, allergies, obesity, kidney failure and stroke.
People do argue as to what are the environmental differences between humans living in developed regions and people living in rural undeveloped regions that presumably lead to the very different incidences of heart disease and other disorders. My hypothesis, of which I am virtually certain, is that a large part of those differences are due to differential NO/NOx physiology due to the presence/absence of a biofilm of ammonia oxidizing bacteria. Many of the major health differences are known to involve pathways mediated by NO signaling. Virtually all of those disorders are characteristic of what would be expected if the NO signaling was skewed in the direction expected by a reduced basal level of NO.
I have instrumental data of NO production in vivo (human) from a biofilm of these bacteria on a subject from spontaneously released ammonia coincident with instrumental measure of a physiological effect that is known to be mediated by NO. I have instrumental measures of NO production in vivo (human) which demonstrate that some of the NO is absorbed. I have instrumental measures that demonstrate long term maintenance of such a biofilm (years) nourished only by natural secretions. My data is only on a single subject, so it is an “anecdote”. That my data is on a single subject does not mean my data is wrong, it only means that it has low statistical significance. The effects I measured are many sigma above background and occurred on many different instances.
Have a look from the gene’s perspective. It lives not just in one human meat container but in a complex ecosystem made up of many containers, not all of them human. Forgive the personification –it’s just a metaphor– but the gene feels as much loyalty to any one container as you feel toward any one of your skin cells.
If some gene containers die early but somehow contribute to greater gene frequency in the entire ecosystem, early container death may become an increasingly popular phenotype.
Ok, “optimum” was a bad word choice. How about “health that is better controlled to what ever phsyiology calls for it to be at the moment because there is better regulation of NO controlled phsyiological pathways because the basal levels of NO/NOx/RSNO species are under better phsyiological control because the normal pathway of NO/NOx production by a surface biofilm of ammonia oxidizing bacteria is not artificially removed”.
Is that better?
I agree that biological creatures have self-regulatory mechanisms that largely maintain favorable internal conditions. However, it’s a mistake to assume that *all* these self-regulatory systems are there to serve your best interests. Some are actually trying to kill you.
Maybe higher NO levels boost the senescence system. Who knows?
You certainly have a interesting, even a fun hypothesis, Daedalus2. It deserves to be true, even if its not.
You seem to be aware of some limitations to it, but not others.
I think the main problem is the way you make certain assertions and then are unable to support them with the evidence required. You need a little more — actually a lot more — of the caution that goes with normal scientific discourse and enquiry.
Perhaps this is getting into trade secrets, but is there bacteriological evidence that the skin flora of non-washing primitive tribes differs from ours? — even quantitatively, as the effects of washing don’t last long, especially in hot weather.
D2:”Linus Pauling was not talking about a physiological intervention.”
Actually, just like you, he thought it was a physiological intervention, and, also like you because of animal experiments strongly suggesting that humans might need vastly more Vitamin C than was normally obtainable in the diet.
I saw the Linus Pauling connection, like qetzal.
If our worst suspicions are correct, and a weak-to-moderately tenable set of hypotheses has you by the tail rather than vice versa, your path is now laid out for you.
You will continue to find evidence that supports your theory and makes it even harder for you to accept that your clinical expectations are wrong, while being less sensitive to anything that might undermine them.
When clinical testing starts you will be reluctant to accept negative results and will try it out this way and with that until you get a positive or two. By that time your patented product will be on the market and testimonial after testimonial will be flooding in.
We are merely urging caution. It takes only a vaguely tenable theory in the hands of an enthusiast to create a quack cure — of anything, even cancer.
I greatly respect daedalus2u’s contributions on other subjects, but I choose not to engage him on the subject of NO. I have to withhold judgment pending better evidence. I think he fully understands that he doesn’t have adequate evidence yet, but his enthusiasm for his subject is too great to restrain.
I am very aware of the potential for quackery. I control the patents, so as long as they are in force, it can’t be used for quackery. I really don’t want to go that route, but going the other path is very difficult without the resources which I don’t have. I am also acutely aware that if I am right, that every day that this treatment is delayed means more pain, suffering and early death for many people. Many, many, many people.
One of the major described differences in the skin of people in the rural undeveloped world is that acne is virtually unknown at any age, even during puberty. The ammonia oxidizing bacteria status of people anywhere has not been reported. I don’t have the facilities to look at it on people, I have looked for them on a number of eukaryotes and found it on many individuals of many species.
The people doing the microbiome have not progressed to testing people in the “wild” yet. They are still foraging for what they think is biodiversity by looking at hundreds of cracks in the pavement of sidewalks (the skin of people in cities who bathe every day) instead of looking at the rain forest (people that never bathe). When I have tried to contact them they do not want to talk to me.
I am not worried about good well-run clinical trials showing negative results. What I am worried about is a false negative due to a badly run trial or because I guess wrong as to the first thing to test this on and don’t get second or third shots. If it works on even a small fraction of the things I think it will work on, it will be hailed as a “wonder drug”. If there is a rational balancing of the risk vs the benefit, even very small benefits will be worth using it for because the risks really are extremely small (I can’t think of any credible ones and neither has anyone else).
I do appreciate that my evidence is not of publishable quality. I have presented a great deal of it at scientific conferences and no one has been able (or willing) to tell me where it is wrong.
I really do appreciate the feedback that I get from people here at SBM. The compulsion to only utilize facts and logic is quite refreshing. Most everywhere else science is driven to a large extent by fashion and everyone is so afraid of proposing something outside the box because is might hurt their career. People (and funding agencies) are extremely “risk averse”, which only means not doing something that might look foolish in hindsight. So they fund and work on things that are fashionable. Other people’s ideas can’t be encouraged because that detracts from the PR surrounding their ideas, and it is that PR that will bring the next rounds of funding.
Right now the fashion is in genes. Genetic research should be done, we need to know what the genes are and how they work, but that will not be the route that solves many of diseases that are the leading causes of morbidity. I am in the process of writing up what will be.
Dr Benway, it is low NO that accelerates senescence.
Low NO increases ROS and ROS damage. High NO prevents ROS damage, quenches superoxide and other radicals, and NO activates telomerase and prevents senescence.
http://jap.physiology.org/cgi/content/full/106/1/333
Cool, daedalus. But “more NO” or “less NO” might not be as easy as you imagine.
I remember when I thought of dopamine as one thing and “more dopamine” and “less dopamine” as meaningful states. Now I understand several dopamine systems. Sometimes a little more in one system means a little less in another.
Harriet,
I respect most of deadalus2u’s contributions as well. But I think intellectual honesty demands that we hold one another to the same standards we expect when we criticize CAM.
How many times has this blog lamented things like the following:
- a single cause for a wide variety of serious diseases
- virtual certainty that a novel intervention will be therapeutic for many conditions, despite the near absence of human data
- insistence that the intervention has zero potential for side effects because it merely restores the body’s natural physiological balance
Those are not the hallmarks of science-based medicine, as I hope you’ll agree. Yet deadalus2u makes such statements here on a regular basis, and is rarely challenged on them.
Quite honestly, besides being hypocritical to SBM’s principles, I think it’s a disservice to deadalus2u. I believe he truly has read a huge amount of literature on NO. I believe there’s a real chance that at least some of his ideas may have therapeutic potential. But I also believe that his irrational fervor (or at least, what I see as an irrational fervor) is almost certain to hinder his chances of successfully testing his ideas.
daedulus,
You don’t have to go to third world countries to find a large population of people who don’t bathe.
quetzal, I’ll worry about daedelus when he’s selling something. Right now he’s just hyperfocused and enthusiastic over the possibilities he sees in his research.
In case you haven’t noticed daedelus is a bit, how you say… neurodiverse. To his credit he is slow to take offense. He doesn’t ragequit in a huff if you question him. He just writes a lot of tl;dr.
I like him.
I have a write-up on NO and the physiology behind CFS. If people send me and email I will be happy to send it to them. The best place to send it is to daedalus4u at yahoo dot com. This was the technical background for an Army proposal for basic research on CFS. The reviewers had only good things to say about the proposal, but it was not funded for reasons that were not explained. I think because they had already decided who they wanted to fund.
I am neurodiverse, I have Asperger’s. I am self-diagnosed after it started getting better after I raised my NO level by applying these bacteria. I didn’t know that I had Asperger’s until after I noticed it started getting better. I only started reading up on autism after I noticed I started getting better and started making the connections to NO physiology. At an autism conference a year ago I was presenting my poster on the connection between NO, functional connectivity and the acute resolution of autism symptoms with fever, and related that I noticed my Asperger’s getting better before I knew I had Asperger’s. An MD in the crowd piped up that “since I have known you, I have noticed your Asperger’s has gotten better”. She is board certified in neurodevelopment. I first met her at an autism conference in 2005 after I had gotten advice from a very very senior NO researcher who had offered to help me, but when I sent him a write-up on the connections between autism and NO, said there were no animal models of autism, his basic research contacts would be no help, I should contact people with clinical experience, which I did.
In retrospect, Asperger’s does explain my whole life, vulnerability to being bullied, extreme proficiency in science, honors graduate MIT, no GF until I was 28. I have a low NO physiology, which I inherited from my mother. I am pretty sure she has Asperger’s too. She died with advanced Alzheimer’s as did both her parents. I am certain it was her low NO physiology that gave her Alzheimer’s (I can send a write-up on Alzheimer’s too). I was on that tract, with many symptoms of low NO before I raised mine, hypertension, the metabolic syndrome, depression, anxiety, PTSD, allergies. All of those got fabulously better when I raised my NO level. Mostly I only figured out the connection to NO physiology after I noticed the symptoms getting better because it was only then that I started reading about the physiology.
I appreciate that these are anecdotes, and I would not believe them had I not experienced them myself, and perhaps not even then if the literature had not provided a NO-based physiological explanation. I appreciate that this does not make compelling evidence. But why most people want to take these anecdotes as evidence that my hypothesis is wrong is not something I understand.
I’m in awe that a single blog entry on this site can draw 139 comments. If I get two or three comments on one of my newsletters, I’m lucky.
I won’t clutter things up further here, but I did want on my website to highlight a fundamental area of disagreement (at least I think it is an area of disagreement).
P.Mean: Are certain CAM therapies undeserving of further study (created 2010-12-01).
http://www.pmean.com/10/UndeservingOfFurtherStudy.html
It’s an interesting discussion.
Steve Simon, http://www.pmean.com
Steve,
Part II of this “redux” series will address that very question. It’s already mostly written, and it addresses many of points that you’ve made in your post today, but I’ll try to find time to respond more specifically to some of them (it already mentions the Emily Rosa experiment as part of a more general topic; our views seem to be converging). BTW, have you been a regular Skeptical Inquirer reader long enough to have seen the article linked here?
KA
Yes, I did read that article, and it was probably in the back of my mind when I wrote the comment “There’s an argument to be made that NCCAM research to date has been pretty much a bust.”
I also found your comments about conflict of interest thought provoking. I do not believe that holding a grant constitutes a conflict of interest (with a very few exceptions), but others do. There’s an article by Dale Hammerschmidt with the provocative title: “When commitments and interests conflict. ‘There’s probably no greater conflict of interest than an NIH grant’.” (http://www.ncbi.nlm.nih.gov/pubmed/7602234) but I disagree with this strongly.
The problem is that the people in the world who have SERIOUS conflicts of interest would like it if it were perceived that EVERYONE has a conflict of interest. Then they could pretend that all these conflicts were of equal concern and they would be allowed back at the table. There would be no faster way to get to this goal than to say that any researcher with a research grant has a conflict of interest.
It is nice that your article is available freely on the web. I wish more of the Skeptical Inquirer and Skeptic magazine articles were that way.
Steve Simon, http://www.pmean.com