Yes, Jacqueline: EBM ought to be Synonymous with SBM
“Ridiculing RCTs and EBM”
Last week Val Jones posted a short piece on her BetterHealth blog in which she expressed her appreciation for a well-known spoof that had appeared in the British Medical Journal (BMJ) in 2003:
Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials
Dr. Val included the spoof’s abstract in her post linked above. The parachute article was intended to be humorous, and it was. It was a satire, of course. Its point was to call attention to excesses associated with the Evidence-Based Medicine (EBM) movement, especially the claim that in the absence of randomized, controlled trials (RCTs), it is not possible to comment upon the safety or efficacy of a treatment—other than to declare the treatment unproven.
A thoughtful blogger who goes by the pseudonym Laika Spoetnik took issue both with Val’s short post and with the parachute article itself, in a post entitled #NotSoFunny – Ridiculing RCTs and EBM.
Laika, whose real name is Jacqueline, identifies herself as a PhD biologist whose “work is split 75%-25% between two jobs: one as a clinical librarian in the Medical Library and one as a Trial Search Coordinator (TSC) for the Dutch Cochrane Centre.” In her post she recalled an experience that would make anyone’s blood boil:
I remember it well. As a young researcher I presented my findings in one of my first talks, at the end of which the chair killed my work with a remark that made the whole room of scientists laugh, but was really beside the point…
This was not my only encounter with scientists who try to win the debate by making fun of a theory, a finding or …people. But it is not only the witty scientist who is to *blame*, it is also the uncritical audience that just swallows it.
I have similar feelings with some journal articles or blog posts that try to ridicule EBM – or any other theory or approach. Funny, perhaps, but often misunderstood and misused by “the audience”.
Jacqueline had this to say about the parachute article:
I found the article only mildly amusing. It is so unrealistic, that it becomes absurd. Not that I don’t enjoy absurdities at times, but absurdities should not assume a life of their own. In this way it doesn’t evoke a true discussion, but only worsens the prejudice some people already have.
Jacqueline argued that two inaccurate prejudices about EBM are that it is “cookbook medicine” and that “RCTs are required for evidence.” Regarding the latter, she made reasonable arguments against the usefulness or ethics of RCTs for “prognostic questions,” “etiologic or harm questions,” or “diagnostic accuracy studies.” She continued:
But even in the case of interventions, we can settle for less than a RCT. Evidence is not present or not, but exists on a hierarchy. RCT’s (if well performed) are the most robust, but if not available we have to rely on “lower” evidence.
BMJ Clinical Evidence even made a list of clinical questions unlikely to be answered by RCT’s. In this case Clinical Evidence searches and includes the best appropriate form of evidence.
- where there are good reasons to think the intervention is not likely to be beneficial or is likely to be harmful;
- where the outcome is very rare (e.g. a 1/10000 fatal adverse reaction);
- where the condition is very rare; [etc., for a total of 6 more categories]
In asserting her view of another inaccurate prejudice about EBM, Jacqueline took Dr. Val and Science-Based Medicine to task:
Informed health decisions should be based on good science rather than EBM (alone).
Dr. Val: “EBM has been an over-reliance on “methodolatry” - resulting in conclusions made without consideration of prior probability, laws of physics, or plain common sense. (….) Which is why Steve Novella and the Science Based Medicine team have proposed that our quest for reliable information (upon which to make informed health decisions) should be based on good science rather than EBM alone.”
Methodolatry is the profane worship of the randomized clinical trial as the only valid method of investigation. This is disproved in the previous sections.
The name “Science Based Medicine” suggests that it is opposed to “Evidence Based Medicine”. At their blog David Gorski explains: “We at SBM believe that medicine based on science is the best medicine and tirelessly promote science-based medicine through discussion of the role of science and medicine.”
While this may apply to a certain extent to quack[ery] or homeopathy (the focus of SBM) there are many examples of the opposite: that science or common sense led to interventions that were ineffective or even damaging, including:
- protective effects of beta-carotene on lung cancer (opposite)
- protective effects of vitamin E on cardiovascular disease (opposite) or certain cancers (neutral)
- protective effects of the prone sleeping on Sudden Infant Death Syndrome (opposite)
- Protective effects of corticosteroids in severe head trauma (opposite)
etcetera.As a matter of fact many side-effects are not foreseen and few in vitro or animal experiments have led to successful new treatments.
At the end it is most relevant to the patient that “it works” (and the benefits outweigh the harms).
Furthermore EBM is not -or should not be- without consideration of prior probability, laws of physics, or plain common sense. To me SBM and EBM are not mutually exclusive.
Jacqueline finished by quoting a few comments that had appeared on the BMJ website after the parachute article. Some of them (not all, I’m happy to report) revealed that their authors lacked a sense of humor. Another argued that “EBM is not RCTs.” Still others argued that RCTs are valuable for precisely the reason illustrated by Jacqueline’s examples listed above: that even some, seemingly safe and effective treatments—based on science or common sense or clinical experience—have eventually been shown, when subjected to RCTs, to behave otherwise. No one at SBM would argue the point.
Science-Based Medicine is Not Opposed to Evidence-Based Medicine
I am confident in asserting that we at SBM are in nearly complete agreement with Jacqueline regarding how EBM ought to be practiced. We are, I’m sure, also in agreement that many objections to EBM are specious. Among these, soundly criticized on this site, are special pleadings and bizarre post-modern arguments. The name “Science-Based Medicine” does not suggest that we are opposed to EBM. What it does suggest is that several of us consider EBM to be incomplete in its gathering of evidence, incomplete in ways that Jacqueline herself touched upon. I explained this in a series of posts at the inception of SBM in 2008,† and I discussed it further at TAM7 last summer. As such, Managing Editor David Gorski invited me to respond to Jacqueline’s article. I am happy to do so because, in addition to clarifying the issues for her, it is important to review the topic periodically: The problems with EBM haven’t gone away, but readers’ memories are finite.
Let me begin by asserting that everyone here agrees that large RCTs are the best tools for minimizing bias in trials of promising treatments, and that RCTs have repeatedly demonstrated their power to refute treatment claims based solely on physiology, animal studies, small human trials, clinical judgment, or whatever. I made the very point in my talk at TAM7, offering the Cardiac Arrhythmia Suppression Trial and the Women’s Health Initiative as examples. We also agree that there are some situations in which RCTs, whether for logistical, ethical, or other reasons, ought not to be used or would not yield useful information even if attempted. Parachutes are an example, but there are subtler ones, e.g., the efficacy of pandemic flu vaccines or whether the MMR vaccine causes autism. As we shall see, however, the list of exceptions offered by Jacqueline and BMJ Clinical Evidence is neither a formal part of EBM nor universally accepted by EBM practitioners.
To reiterate: The most important contribution of EBM has been to formally emphasize that even a high prior probability is not always sufficient to establish the usefulness of a treatment—parachutes being exceptions.
EBM’s Scientific Blind Spot
Now, however, we come to an important problem with EBM, a problem not merely of misinterpretations of its tenets (although such are common), but of the tenets themselves. Although a reasonably high prior probability may not be a sufficient basis for incorporating a treatment into general use, it is a necessary one. It is, moreover, a necessary basis for seriously considering such a treatment at all; that is, for both scientific and ethical reasons it is a prerequisite for performing a randomized, controlled human trial. Rather than explain these points here and now, I ask you, Dear Reader, to indulge me by following this link to a post in which I have already done so in some detail. I’ll wait here patiently.
……………….
Are you back? OK. Now you know that we at SBM are in total agreement with Jacqueline that EBM “should not be without consideration of prior probability, laws of physics, or plain common sense,” and that SBM and EBM should not only be mutually inclusive, they should be synonymous. You also know, however, that Jacqueline was mistaken to claim that EBM already conforms to those ideals. It does not, and its failure to do so is written right into its Levels of Evidence scheme—the exceptions that she offered, including those quoted from BMJ Clinical Evidence, notwithstanding. You know all of this because you’ve now seen several examples (there are many more) from that wellspring of EBM reviews, Jacqueline’s own Cochrane Collaboration. (There is another, more subtle reason for prior probability being overlooked in EBM literature, but it is an optional exercise for the purposes of today’s discussion).
EBM and Unintended Mischief
The problems caused by EBM’s scientific blind spot are not limited to the embarrassment of Cochrane reviews suggesting potential clinical value for inert treatments that have been definitively refuted by basic science, although that would be sufficient to argue for EBM reform. The Levels of Evidence scheme has resulted in dangerous or unpleasant treatments being wished upon human subjects in the form of RCTs, cohort studies, or case series even when existing clinical or scientific evidence should have been more than satisfactory to put such claims to rest. The Trial to Assess Chelation Therapy (TACT)—the largest, most expensive, and most unethical trial yet funded by the NCCAM—was originally justified by these words in an editorial in the American Heart Journal in 2000, co-authored by Gervasio Lamas, who would later become the TACT Principal Investigator:
The modern standard for accepting any therapy as effective requires that there be scientific evidence of safety and efficacy in a fair comparison of the new therapy to conventional care. Such evidence, when widely disseminated, leads to changes in clinical practice, ultimately benefitting patients. However, the absence of a clinical trial does not disprove potential efficacy, and a well-performed but too small “negative” trial may not have the power to exclude a small or moderate benefit of therapy. In other words, the absence of evidence of efficacy does not constitute evidence of absence of efficacy. These concepts constitute the crux of the lingering controversy over chelation therapy…
Such an argument, with its obvious appeal to the formal tenets of EBM, was made and accepted by the NIH in spite of overwhelming evidence against the safety and effectiveness of Na2EDTA chelation treatments for atherosclerotic vascular disease, including the several “small” disconfirming RCTs, comprising approximately 270 subjects, to which Dr. Lamas alluded. It was also accepted in spite of its violating both the Helsinki Declaration and the NIH’s own policy stipulating that preliminary RCTs should demonstrate efficacy prior to a Phase III trial being performed.
A 2006 Cochrane Review of Laetrile for cancer would, if its recommendations were realized, stand the rationale for RCTs on its head:
The most informative way to understand whether Laetrile is of any use in the treatment of cancer, is to review clinical trials and scientific publications. Unfortunately no studies were found that met the inclusion criteria for this review.
Authors’ conclusions
The claim that Laetrile has beneficial effects for cancer patients is not supported by data from controlled clinical trials. This systematic review has clearly identified the need for randomised or controlled clinical trials assessing the effectiveness of Laetrile or amygdalin for cancer treatment.
Why does this stand the rationale for RCTs on its head? A definitive case series led by the Mayo Clinic in the early 1980s had overwhelmingly demonstrated, to the satisfaction of all reasonable physicians and biomedical scientists, that not only were the therapeutic claims for Laetrile baseless, but that the substance is dangerous. The subjects did so poorly that there would have been no room for a meaningful advantage in outcome with active treatment compared to placebo or standard treatment—as we have recently seen in another trial of a quack cancer treatment. The Mayo case series “closed the book on Laetrile,” the most expensive health fraud in American history at the time, only to have it reopened more than 20 years later by well-meaning Cochrane reviewers who seemed oblivious of the point of an RCT.
A couple of years ago I was surprised to find that one of the authors of that review was Edzard Ernst, a high-powered academic who over the years has undergone a welcomed transition from cautious supporter to vocal critic of much “CAM” research and many “CAM” methods. He is now a valuable member of our new organization, the Institute for Science in Medicine, and we are very happy to have him. I believe that his belated conversion to healthy skepticism was due, in large part, to his allegiance to the formal tenets of EBM. I recommend a short debate published in 2003 in Dr. Ernst’s Focus on Alternative and Complementary Therapies (FACT), pitting Jacqueline’s countryman Cees Renckens against Dr. Ernst himself. Dr. Ernst responded to Dr. Renckens’s plea to apply science to “CAM” claims with this statement:
In the context of EBM, a priori plausibility has become less and less important. The aim of EBM is to establish whether a treatment works, not how it works or how plausible it is that it may work. The main tool for finding out is the RCT. It is obvious that the principles of EBM and those of a priori plausibility can, at times, clash, and they often clash spectacularly in the realm of CAM.
I’ve discussed that debate before on SBM, and I consider it exemplary of what is wrong with how EBM weighs the import of prior probability. Dr. Ernst, if you are reading this, I’d be interested to know whether your views have changed. I hope that you no longer believe that human subjects ought to be submitted to a randomized, controlled trial of Laetrile!
When RCTs Mislead
Finally, for the purposes of today’s discussion, let me reiterate another point that must be considered in the context of establishing, via the RCT, whether a treatment works: When RCTs are performed on ineffective treatments with low prior probabilities, they tend not to yield merely ‘negative’ findings, as most physicians steeped in EBM would presume; they tend, in the aggregate, to yield equivocal findings, which are then touted by advocates as evidence favoring such treatments, or at the very least favoring more trials—a position that even skeptical EBM practitioners have little choice but to accept, with no end in sight. Numerous such examples have been discussed on this website.
The first sentence that I ever posted on SBM, a quotation from homeopath David Reilly, was a perfect illustration of this misunderstanding:
Either homeopathy works or controlled trials don’t!
Dr. Reilly was correct, of course, but not in the way that he supposed. If there is anything that the history of parapsychology can teach the biomedical world, it is the point just made: human RCTs, as good as they are at minimizing bias or chance deviations from population parameters, cannot ever be expected to provide, by themselves, objective measures of truth. There is still ample room for erroneous conclusions. Without using broader knowledge (science) to guide our thinking, we will plunge headlong into a thicket of errors—exactly as happened in parapsychology for decades and is now being repeated by its offspring, “CAM” research.
Conclusion
These are the reasons that we call our blog “Science-Based Medicine.” It is not that we are opposed to EBM, nor is it that we believe EBM and SBM to be mutually exclusive. On the contrary: EBM is currently a subset of SBM, because EBM by itself is incomplete. We eagerly await the time that EBM considers all the evidence and will have finally earned its name. When that happens, the two terms will be interchangeable.
…
† The Prior Probability, Bayesian vs. Frequentist Inference, and EBM Series:
1. Homeopathy and Evidence-Based Medicine: Back to the Future Part V
2. Prior Probability: The Dirty Little Secret of “Evidence-Based Alternative Medicine”
3. Prior Probability: the Dirty Little Secret of “Evidence-Based Alternative Medicine”—Continued
4. Prior Probability: the Dirty Little Secret of “Evidence-Based Alternative Medicine”—Continued Again
5. Yes, Jacqueline: EBM ought to be Synonymous with SBM
6. The 2nd Yale Research Symposium on Complementary and Integrative Medicine. Part II
7. H. Pylori, Plausibility, and Greek Tragedy: the Quirky Case of Dr. John Lykoudis
10. Of SBM and EBM Redux. Part I: Does EBM Undervalue Basic Science and Overvalue RCTs?
11. Of SBM and EBM Redux. Part II: Is it a Good Idea to test Highly Implausible Health Claims?
12. Of SBM and EBM Redux. Part III: Parapsychology is the Role Model for “CAM” Research
13. Of SBM and EBM Redux. Part IV: More Cochrane and a little Bayes
14. Of SBM and EBM Redux. Part IV, Continued: More Cochrane and a little Bayes
15. Cochrane is Starting to ‘Get’ SBM!
16. What is Science?
Zoe,
You must come to Australia.
Referred directly by a GP, you would pay only $265 (USD). Referred via the local public hospital outpatient’s department, it would cost you absolutely nothing (there are two drawbacks though: there will be a delay of a couple of months; the orthopaedic surgeon may not consider it necessary for you to have an MRI).
Also, no one in Australia is uninsured unless they actually fail to register with Medicare. Registration is free, and Medicare is totally 100% publically funded (ie paid out of taxes by those who can afford it but don’t take out private health insurance).
I ask you, is there a better system anywhere?
JMB,
I was just alerted to the following link in another thread. It is mainly about the influence of drug companies on doctor’s prescribing habits (as viewed by Dr. Mark Crislip), but I think parts of it are relevant here:
http://www.sciencebasedmedicine.org/?p=60
“It is odd isn’t it? Large numbers of quality studies published in the best peer review journals consistently showing the same or similar effect and no contradictory studies. Despite the emphasis on evidence-based medicine, the entire literature is dismissed as not relevant because personal experience suggests that the studies are wrong.”
“Curiouser and Curiouser. Coherent arguments as to the validity and scope of the literature are met with denial but never a critique of the primary literature. The facts of the research are never argued. The only argument is personal experience or blanket denial. Despite the published literature, practice continues the same, untouched by the facts.”
“the three most dangerous words in medicine (”In my experience”)”
Of course, that is only the author’s view but…
regards,
BillyJoe
“Zoe,
You must come to Australia.
Referred directly by a GP, you would pay only $265 (USD). Referred via the local public hospital outpatient’s department, it would cost you absolutely nothing (there are two drawbacks though: there will be a delay of a couple of months; the orthopaedic surgeon may not consider it necessary for you to have an MRI).”
Yes, I only had to wait four days for my non-emergency MRI.
BillyJoe wrote:
“Let’s face it, JMB, if it costs 7 times as much to get an MRI, health care is the US of A is #v<k#d, whatever reasons you may come up with for the difference."
I agree. That is why doctors in the US favor healthcare reform (but disagree on what is needed). Specific reforms could be passed to lower the cost of healthcare addressing the type of issues I discussed, but that is not happening. What is being emphasized in the current healthcare reform bills is the idea of expanding insurance coverage, not measures to reduce the cost of healthcare. It is true the cost of healthcare can be reduced by insisting on EBM, but from the way it is being implemented in the US, it may result in the lack of funding for potentially effective diagnostic tests or treatments. We could go whole hog into a socialized healthcare system, but based on what I have seen in the Veterans Administration system, I doubt a socialized healthcare system run by our government would be as good as the ones in other countries. Lobbyists are too powerful in this country, bureaucracy is too big here.
Zoe237 wrote:
"So it’s basically impossible to find out the true cost of something, especially if you are uninsured. It’s a huge racket."
Absolutely it is a racket! Anybody who tells you that the free market forces are working for the uninsured patient hasn't a clue of how things work. Believe it or not, the government used to discourage healthcare providers from publishing charges, because it would lead to price fixing (according to a government bureaucrat). I think the current healthcare reform bills may actually include provisions to publish charges. President Bush actually campaigned to get hospitals to publish charges, but was unsuccessful. I wish the current healthcare reform bills would make all providers publish their "usual and customary fees for service" for the previous five years, so the public could see who was really jacking up prices and screwing patients without insurance, just so they could sell accounts receivable to collection agencies at a higher price. Price controls would be the fastest method to bring down the cost of medical care for the uninsured.
Plonit wrote:
"Since cost per scan is partly made up of Capital Costs/Scans, oversupply may add to the cost (and cancel out benefit of competition between hospitals offering MRI)."
I think that is a complex economic issue. The generous reimbursement for MRIs lead to a profitable MRI business. As the supply of MRI scanners (and CT scanners) outstripped the demand based on necessary tests, the price per test should have dropped based on the law of supply and demand. There was a short time period in which the cost began to drop. However, because of the lack of regulation, too many healthcare entrepreneurs figured out how to increase utilization, The highest concentration of diagnostic imaging facilities had the highest rate of utilization. There were numerous cities in the United States that had more MRI facilities in five square miles of high income areas than the entire nation of Canada. Now if Congress had stood up to the lobbyists, this
"entrepreneurial spirit" could have been nipped in the bud 15 years ago. Basically if congress had outlawed physicians from making a profit from scans they ordered, this wouldn't have happened. So the capital cost of the MRI scanners is only a drop in the bucket compared to overutilzation costs in our system. In a socialized medical system, the capital cost may be a bigger factor in driving up the healthcare cost.
It really was the outpatient imaging centers giving competition to the hospitals. When free of anticompetitive bias, the outpatient imaging center would usually have a lower price than the hospital. Ironically, that competition may be reduced in some of the more obscure provisions of the healthcare reform bills.
BillyJoe wrote:
"In fact, you didn't even answer that question. We are not talking about diagnosis here, we are talking about treatments."
That is why the original moniker of "Computer Aided Diagnosis" morphed into "Medical Decision Making" The same decision analytic approaches (predominately based on Bayes strategies) could be applied to treatment decisions as well as diagnostic decisions. Nowadays, the moniker "computer aided diagnosis" is usually applied only to systems of automated pattern recognition. Any medical decision is tested by its future outcomes, so my observations apply to both diagnosis and treatment decisions. Now comparative effectiveness of different treatments and diagnostic tests is a different question, for which different methods are applied.
I didn't mean to imply the RCT's are useless. They remain the most important underlying scientific method for medicine. If you want to determine that a treatment or diagnostic test is effective, the RCT is the best way to determine effectiveness, if it is feasible.
But I am trying to point out the difference in the task of proving whether a diagnostic test or treatment is effective, and the task of making medical decisions for a patient. RCT's ignore variations in the population of patients, and in the skills of the healthcare providers. A properly designed RCT minimizes the impact of those variations on the outcomes by randomization of the patients, and incorporation of multiple medical institutions in the trial. As a healthcare provider facing an individual patient, you do not have to ignore all of those individual variations, or the variations in outcomes that may be associated with the skills of the providers.
On another level of application of EBM versus the broader set of scientific method of SBM. When the quality of an RCT is judged, should it be judged solely by a researcher with extensive training of experimental design and statistical analysis? Or should someone with clinical experience be involved in the judgment of the quality of the trial? A case in point is the Canadian National Breast Screening Study. The EBM approach is well documented on the USPSTF website. I attended a presentation by a leading mammography researcher about 15 years ago. As I recall, he criticized the quality of mammography and mammographic interpretation when asked to review it at the outset of the trial. He was not the first mammogram expert to be invited to participate. Others had refused to participate because of the poor quality. Several of the facilities in the trial were performing only single view xerograms for the screening mammogram. This expert offered to go out and train the facilities on how to perform the exam correctly, but the principle investigators refused because their goal was to determine whether mammography as currently being performed in Canada (the 1980's) should be recommended as a screening tool to prevent breast cancer deaths. They eventually found an expert to participate in the trial. The experts comments were that there were problems in the quality of mammography, but that the mammographic quality improved in the course of the trial. The CNBSS trial answer was that screening mammography as performed in the study was ineffective for reducing breast cancer deaths below the age of 60.
So the next question is, given the information about the difference in quality of mammography as judged by a clinical expert, would you accept the CNBSS RCT as a better basis for clinical decision? Of course this refers back to the determination of what represents a "properly conducted RCT". So there may not be much difference between what we are talking about, if you are using the experience of a clinical expert to judge whether the trial was properly conducted. In many advisory panels for EBM, the criteria for a properly conducted RCT focuses on the experimantal design, and not on the technical issues (such as mammographic quality). Consequently, the data of the CNBSS trial was included in the computer simulations by the USPSTF to estimate the figure that 1904 women aged 40 to 50 would need to be screened for ten years to prevent one breast cancer death. Would you rely on this estimate in making recommendations to women, or would you guess that the figure is closer to the estimated 1339 women screened in the age group 50 – 60 to prevent one breast cancer death?
So the bottom line I am trying to draw for EBM? It is the most important foundation we have for determining effectiveness of treatments and diagnostic tests, but the use of EBM needs to be tempered by the broader set of scientific methods of SBM. My conclusion about homeopathy would be that the evidence shows that it relies on placebo effect, not that the evidence is inconclusive. I think that is agreeing with the author of this article.
I diverged from the article in discussion to talk about potential pitfalls in the application of EBM in US healthcare reform and by insurance companies, in which reducing unnecessary procedures based on science can become rationing of effective procedures.
I also diverged from the central theme by arguing (unsuccessfully) that the optimum scientific method for making a medical decision about diagnosis or treatment maximizes the use of all available information about the patient and the treatment or test. Ignoring the value of clinical experience reduces the amount of information that may be utilized to make a decision.
It has been an interesting discussion. Thank you very much. I have learned much. My vacation time is over, back to work.
I agree that many physicians will decide to use a particular prescription because the pharmaceutical company has paid for their golf vacation. Most will give the patient what they want in spite of RCT’s and comparative effectiveness research (a recent government policy change will actually increase the pressure to give the patient what they want, rather than what they need based on EBM or SBM).
That is why I draw the distinction between the good clinician, and the bad clinician. Allowing doctors to make decisions based on clinical experience requires oversight, just as oversight of an EBM advisory panel is required.
Maybe one solution would be to require a doctor who changes their practice habits in favor of a sponsor of a meeting (like a pharmaceutical company) to pay for and attend an education seminar about scientific assessment of information (far from any beach, casino, cruise ship, golf course, ski area, etc.).
I couldn’t refrain from one more post trying to convince people that there is a difference in the task of proving a medical intervention is effective, and the task of making a medical decision about an intervention for a patient.
If we are to prove that a medical intervention is effective, we must have sufficient statistical power to show whether an intervention makes a difference in a prospective trial that can be reproduced. Consequently, we use a simple model to separate the experimental and control groups. Is we use too complex of a model, it is hard to find a sufficient number of subjects for the study. For example, if we design an RCT to test the effectiveness of screening mammography, we would simply take a large number of women of a certain age for the subjects of the experiment. We would develop some simple criteria for inclusion and exclusion such as no personal history of breast cancer. Since we know that there is a variation in breast cancer risk based on age, ethnicity, and family history, we would randomize the women within the age groups we have picked, so that there should be no difference in the average ages of the groups, distribution of ethnicity, or incidence of women with positive family histories, between the experimental and control group. Then we perform the study over a number of years, and use various methods of determining either the significance of the difference between the two groups, or estimate the relative risk with it’s confidence interval. Determination of effectiveness is now completed.
Now if we wanted to maximize the outcome for an individual patient (given that we have already determined that the intervention is effective), we would take a slightly different approach. We would construct a more complex model of the process. We would consider the evidence that breast cancer is related to the age of the patient, the number of years of exposure to estrogen, genetics that may be based on either genetic tests or family history, the distribution of the rate of growth of various tumors, the weight of the subject, and the tendency to metastasize based on different sizes of the tumor. Furthermore, we would consider the quality of the mammographic exam, the skill of the interpreter, the density of the breast tissue, and the success of treatment of different types of tumors at various stages. We would record all of those factors in the subjects of the experiment. Then after completing the experiment, we would have some estimates of the conditional probabilities that are factors in our probabilistic model.
Now in the decision for the patient, we could either base our recommendation for the patient on the classic RCT, or we could evaluate factors such as age of menarche, age of menopause, age of first pregnancy, family history, availability of digital mammography, interpreter skill to make a recommendation about when to start mammography, how frequently to have it performed, where to have it done, the likelyhood that it will have a benefit, and the likelyhood of a false positive. Since age is not a perfect predictor of breast density, the patient might undergo a baseline exam to determine breast density (then she would also be in a better position to judge the discomfort as well). Then the best decision for that patient can be reached. We would use our more complex model to arrive at the predictions of likelyhood. Furthermore, if there was a dramatic breakthrough in the treatment of breast cancer, we could use the model to change recommendations for screening studies, rather than waiting 13 years for another RCT to be completed.
Now, in this particular example, the incidence of disease is low in any healthy population except for the elderly. so clinical experience is less likely to lead to functional conditional probabilities, and greater data gathering is necessary for this approach. However, population databases are becoming more complete (an advantage of socialized medicine), and reasonable measure of such probabilities are becoming available.
Going back to my original suggestion that clinical experience is important in medical decisions. The “good” clinician would have repeated clinical experience with dealing with patients suffering from a disease process. They would have a concept of the disease process (a complex model compared to the simplified model used in an RCT). They would have observation skills (physical exam findings), and interviewing skills (obtaining an accurate history), as well as skills of rapport and empathy. They would assess the factors in their internalized model of the disease process, to make a recommendation for the patient. Their internalized model of the disease process can be updated by published RCT, clinical series, and personal observations.
In my experience studying medical decision making, we could identify many clinicians using that approach that could beat our methods of discriminant analysis. Depending on how complete the measurements of the factors in the models, and how dependent the models were on the observational skills of the physician, either a small percentage or a large percentage of clinicians could beat the decision from the computer program.
The argument of what scientific methods that should be applied in different scenarios is not unique to medicine. These are quotes from a lecture published on the internet titled, “Frequentist vs. Bayesian vs. Probabilist” from the quantum optics and biophysics group in the Applied Physics Department at Stanford University. I would assume the author is the professor, Hideo Mabuchi.
1.However, I think the role of the physicist
modeling nature is to maximize his winnings in the game of “guess what nature will do”.
2.The lesson is that we should not trust the statistical inference unless we are convinced that the underlying probabilistic model is realistic.
Applied physics is much more specific in developing models than medical science. Medical models of disease may become more formalized (in mathematical description) based on successes of advances in medical genetics.
Now not all clinicians will reach that which I would call a good clinician, it has much to do with problem solving ability and powers of observation (not just academic abilities). Furthermore, no clinician can have universal experience. Finally, clinical load can be overwhelming, and errors can be make in communication of status and results, and doctors can make decisions for nefarious reasons (thanking the pharmaceutical company for the golf trip). Consequently, an approach based on EBM (I won’t call it algorithmic, but that would still be my classification based on medical decision making) can improve practice of medicine. Doctors should be allowed some leeway for clinical experience, because if they are good, then they may have even better results. Whether or not deviations from EBM yield better results can be retrospectively reviewed. If the doctor deviating from EBM has no better results but higher costs, or has worse results, then remedial action can be undertaken.
So in summary, the best scientific method fro making a decision about an intervention is more like the problem of predicting the weather, rather than determining if there is cause and effect. You wouldn’t base the prediction of weather on Feb 14 from the average of the last ten years. You would base it on weather models and current conditions.
Do the SBM advocates tell us how to obtain a consensus prior on the probability of treatment efficacy, or are there as many versions of SBM as there are practitioners, and if that’s the case aren’t quacks with high priors on their quackery justified (by SBM) in administering it in some cases?
I’m suggesting some codification might be useful, otherwise it’s just the wild west.
@rork:
There’s certainly plenty of room for disagreement on the best way to determine prior probability. But not to the extent of giving acupuncture or homeopathy a significant prior probability. There’s not THAT much wiggle room.
It’s kind of like how arguing the details of murder vs. self-defense in a case where the dead person broke into the defendant’s house, shot him in the arm, and the defendant shot back and killed him. Sure there’s room for arguing exactly where the proper line lies, but individual cases aren’t necessarily anywhere near the line.
I would suggest three sources of authoritative priors to stem the wild west approach of “in my experience”.
1> Probabilistic models based on medical science. Few doctors don’t have some idea in their head about what causes a disease, what the natural course of the disease is, and what may alter the course of the disease. Mostly, it is the physicians in academic positions who have the ability to translate a medical model of disease into a mathematical model suitable for probability calculations. Mathematical models can be tested by prospective studies that are not as involved as RCT’s.
2> Good clinicians identified by outcome measures become sources for priors. This would not be limited to academic medicine.
3> Large databases such as those maintained by socialized medicine can become sources. I guess this mixes the frequentist and Bayesian approach.
These are not mutually exclusive sources of information.
The probabilistic models will tend to cut down on quackery. The probabilistic models would also be modified by the results of RCT’s and other arguments in the medical literature.