Articles

110 thoughts on “Yes, Jacqueline: EBM ought to be Synonymous with SBM

  1. Zoe237 says:

    “By the way, the average reimbursements for an MRI of the knee is somewhere between $600 and $1000. The only patients that pay $2000 are those without insurance who have an MRI at a more expensive facility.”

    That is true. My bill says $1,971 for the total, $328 paid for by the insurance company, “minus participating provider savings” $1,143, and I paid $500 (as my deductible). Blue Cross Blue Shield, as my insurer, supposedly negotiated a lower rate in their contract with the hospital. That’s only the cost if I don’t have health insurance, which is around 46 million people, or 15% of the population (!!!). Basically, with many medical services, hospitals and doctors take whatever the insurance company will give them for payments. Medicaid, as the largest insurer, generally sets the prices for various procedures and treatments. At least, that’s how I understand it. So it’s basically impossible to find out the true cost of something, especially if you are uninsured. It’s a huge racket.

    The problem with using clinician experience over that of research evidence uncontrolled confounders, normal human biases. It’s basically the plural of anecdotes.

  2. BillyJoe says:

    Zoe,

    You must come to Australia.

    Referred directly by a GP, you would pay only $265 (USD). Referred via the local public hospital outpatient’s department, it would cost you absolutely nothing (there are two drawbacks though: there will be a delay of a couple of months; the orthopaedic surgeon may not consider it necessary for you to have an MRI).

    Also, no one in Australia is uninsured unless they actually fail to register with Medicare. Registration is free, and Medicare is totally 100% publically funded (ie paid out of taxes by those who can afford it but don’t take out private health insurance).

    I ask you, is there a better system anywhere?

  3. BillyJoe says:

    JMB,

    I was just alerted to the following link in another thread. It is mainly about the influence of drug companies on doctor’s prescribing habits (as viewed by Dr. Mark Crislip), but I think parts of it are relevant here:

    http://www.sciencebasedmedicine.org/?p=60

    “It is odd isn’t it? Large numbers of quality studies published in the best peer review journals consistently showing the same or similar effect and no contradictory studies. Despite the emphasis on evidence-based medicine, the entire literature is dismissed as not relevant because personal experience suggests that the studies are wrong.”

    “Curiouser and Curiouser. Coherent arguments as to the validity and scope of the literature are met with denial but never a critique of the primary literature. The facts of the research are never argued. The only argument is personal experience or blanket denial. Despite the published literature, practice continues the same, untouched by the facts.”

    “the three most dangerous words in medicine (”In my experience”)”

    Of course, that is only the author’s view but…

    regards,
    BillyJoe

  4. Zoe237 says:

    “Zoe,

    You must come to Australia.

    Referred directly by a GP, you would pay only $265 (USD). Referred via the local public hospital outpatient’s department, it would cost you absolutely nothing (there are two drawbacks though: there will be a delay of a couple of months; the orthopaedic surgeon may not consider it necessary for you to have an MRI).”

    Yes, I only had to wait four days for my non-emergency MRI.

  5. JMB says:

    BillyJoe wrote:

    “Let’s face it, JMB, if it costs 7 times as much to get an MRI, health care is the US of A is #v<k#d, whatever reasons you may come up with for the difference."

    I agree. That is why doctors in the US favor healthcare reform (but disagree on what is needed). Specific reforms could be passed to lower the cost of healthcare addressing the type of issues I discussed, but that is not happening. What is being emphasized in the current healthcare reform bills is the idea of expanding insurance coverage, not measures to reduce the cost of healthcare. It is true the cost of healthcare can be reduced by insisting on EBM, but from the way it is being implemented in the US, it may result in the lack of funding for potentially effective diagnostic tests or treatments. We could go whole hog into a socialized healthcare system, but based on what I have seen in the Veterans Administration system, I doubt a socialized healthcare system run by our government would be as good as the ones in other countries. Lobbyists are too powerful in this country, bureaucracy is too big here.

    Zoe237 wrote:
    "So it’s basically impossible to find out the true cost of something, especially if you are uninsured. It’s a huge racket."

    Absolutely it is a racket! Anybody who tells you that the free market forces are working for the uninsured patient hasn't a clue of how things work. Believe it or not, the government used to discourage healthcare providers from publishing charges, because it would lead to price fixing (according to a government bureaucrat). I think the current healthcare reform bills may actually include provisions to publish charges. President Bush actually campaigned to get hospitals to publish charges, but was unsuccessful. I wish the current healthcare reform bills would make all providers publish their "usual and customary fees for service" for the previous five years, so the public could see who was really jacking up prices and screwing patients without insurance, just so they could sell accounts receivable to collection agencies at a higher price. Price controls would be the fastest method to bring down the cost of medical care for the uninsured.

    Plonit wrote:
    "Since cost per scan is partly made up of Capital Costs/Scans, oversupply may add to the cost (and cancel out benefit of competition between hospitals offering MRI)."

    I think that is a complex economic issue. The generous reimbursement for MRIs lead to a profitable MRI business. As the supply of MRI scanners (and CT scanners) outstripped the demand based on necessary tests, the price per test should have dropped based on the law of supply and demand. There was a short time period in which the cost began to drop. However, because of the lack of regulation, too many healthcare entrepreneurs figured out how to increase utilization, The highest concentration of diagnostic imaging facilities had the highest rate of utilization. There were numerous cities in the United States that had more MRI facilities in five square miles of high income areas than the entire nation of Canada. Now if Congress had stood up to the lobbyists, this
    "entrepreneurial spirit" could have been nipped in the bud 15 years ago. Basically if congress had outlawed physicians from making a profit from scans they ordered, this wouldn't have happened. So the capital cost of the MRI scanners is only a drop in the bucket compared to overutilzation costs in our system. In a socialized medical system, the capital cost may be a bigger factor in driving up the healthcare cost.

    It really was the outpatient imaging centers giving competition to the hospitals. When free of anticompetitive bias, the outpatient imaging center would usually have a lower price than the hospital. Ironically, that competition may be reduced in some of the more obscure provisions of the healthcare reform bills.

    BillyJoe wrote:
    "In fact, you didn't even answer that question. We are not talking about diagnosis here, we are talking about treatments."

    That is why the original moniker of "Computer Aided Diagnosis" morphed into "Medical Decision Making" The same decision analytic approaches (predominately based on Bayes strategies) could be applied to treatment decisions as well as diagnostic decisions. Nowadays, the moniker "computer aided diagnosis" is usually applied only to systems of automated pattern recognition. Any medical decision is tested by its future outcomes, so my observations apply to both diagnosis and treatment decisions. Now comparative effectiveness of different treatments and diagnostic tests is a different question, for which different methods are applied.

    I didn't mean to imply the RCT's are useless. They remain the most important underlying scientific method for medicine. If you want to determine that a treatment or diagnostic test is effective, the RCT is the best way to determine effectiveness, if it is feasible.

    But I am trying to point out the difference in the task of proving whether a diagnostic test or treatment is effective, and the task of making medical decisions for a patient. RCT's ignore variations in the population of patients, and in the skills of the healthcare providers. A properly designed RCT minimizes the impact of those variations on the outcomes by randomization of the patients, and incorporation of multiple medical institutions in the trial. As a healthcare provider facing an individual patient, you do not have to ignore all of those individual variations, or the variations in outcomes that may be associated with the skills of the providers.

    On another level of application of EBM versus the broader set of scientific method of SBM. When the quality of an RCT is judged, should it be judged solely by a researcher with extensive training of experimental design and statistical analysis? Or should someone with clinical experience be involved in the judgment of the quality of the trial? A case in point is the Canadian National Breast Screening Study. The EBM approach is well documented on the USPSTF website. I attended a presentation by a leading mammography researcher about 15 years ago. As I recall, he criticized the quality of mammography and mammographic interpretation when asked to review it at the outset of the trial. He was not the first mammogram expert to be invited to participate. Others had refused to participate because of the poor quality. Several of the facilities in the trial were performing only single view xerograms for the screening mammogram. This expert offered to go out and train the facilities on how to perform the exam correctly, but the principle investigators refused because their goal was to determine whether mammography as currently being performed in Canada (the 1980's) should be recommended as a screening tool to prevent breast cancer deaths. They eventually found an expert to participate in the trial. The experts comments were that there were problems in the quality of mammography, but that the mammographic quality improved in the course of the trial. The CNBSS trial answer was that screening mammography as performed in the study was ineffective for reducing breast cancer deaths below the age of 60.

    So the next question is, given the information about the difference in quality of mammography as judged by a clinical expert, would you accept the CNBSS RCT as a better basis for clinical decision? Of course this refers back to the determination of what represents a "properly conducted RCT". So there may not be much difference between what we are talking about, if you are using the experience of a clinical expert to judge whether the trial was properly conducted. In many advisory panels for EBM, the criteria for a properly conducted RCT focuses on the experimantal design, and not on the technical issues (such as mammographic quality). Consequently, the data of the CNBSS trial was included in the computer simulations by the USPSTF to estimate the figure that 1904 women aged 40 to 50 would need to be screened for ten years to prevent one breast cancer death. Would you rely on this estimate in making recommendations to women, or would you guess that the figure is closer to the estimated 1339 women screened in the age group 50 – 60 to prevent one breast cancer death?

    So the bottom line I am trying to draw for EBM? It is the most important foundation we have for determining effectiveness of treatments and diagnostic tests, but the use of EBM needs to be tempered by the broader set of scientific methods of SBM. My conclusion about homeopathy would be that the evidence shows that it relies on placebo effect, not that the evidence is inconclusive. I think that is agreeing with the author of this article.

    I diverged from the article in discussion to talk about potential pitfalls in the application of EBM in US healthcare reform and by insurance companies, in which reducing unnecessary procedures based on science can become rationing of effective procedures.

    I also diverged from the central theme by arguing (unsuccessfully) that the optimum scientific method for making a medical decision about diagnosis or treatment maximizes the use of all available information about the patient and the treatment or test. Ignoring the value of clinical experience reduces the amount of information that may be utilized to make a decision.

    It has been an interesting discussion. Thank you very much. I have learned much. My vacation time is over, back to work.

  6. JMB says:

    I agree that many physicians will decide to use a particular prescription because the pharmaceutical company has paid for their golf vacation. Most will give the patient what they want in spite of RCT’s and comparative effectiveness research (a recent government policy change will actually increase the pressure to give the patient what they want, rather than what they need based on EBM or SBM).

    That is why I draw the distinction between the good clinician, and the bad clinician. Allowing doctors to make decisions based on clinical experience requires oversight, just as oversight of an EBM advisory panel is required.

    Maybe one solution would be to require a doctor who changes their practice habits in favor of a sponsor of a meeting (like a pharmaceutical company) to pay for and attend an education seminar about scientific assessment of information (far from any beach, casino, cruise ship, golf course, ski area, etc.).

  7. JMB says:

    I couldn’t refrain from one more post trying to convince people that there is a difference in the task of proving a medical intervention is effective, and the task of making a medical decision about an intervention for a patient.

    If we are to prove that a medical intervention is effective, we must have sufficient statistical power to show whether an intervention makes a difference in a prospective trial that can be reproduced. Consequently, we use a simple model to separate the experimental and control groups. Is we use too complex of a model, it is hard to find a sufficient number of subjects for the study. For example, if we design an RCT to test the effectiveness of screening mammography, we would simply take a large number of women of a certain age for the subjects of the experiment. We would develop some simple criteria for inclusion and exclusion such as no personal history of breast cancer. Since we know that there is a variation in breast cancer risk based on age, ethnicity, and family history, we would randomize the women within the age groups we have picked, so that there should be no difference in the average ages of the groups, distribution of ethnicity, or incidence of women with positive family histories, between the experimental and control group. Then we perform the study over a number of years, and use various methods of determining either the significance of the difference between the two groups, or estimate the relative risk with it’s confidence interval. Determination of effectiveness is now completed.

    Now if we wanted to maximize the outcome for an individual patient (given that we have already determined that the intervention is effective), we would take a slightly different approach. We would construct a more complex model of the process. We would consider the evidence that breast cancer is related to the age of the patient, the number of years of exposure to estrogen, genetics that may be based on either genetic tests or family history, the distribution of the rate of growth of various tumors, the weight of the subject, and the tendency to metastasize based on different sizes of the tumor. Furthermore, we would consider the quality of the mammographic exam, the skill of the interpreter, the density of the breast tissue, and the success of treatment of different types of tumors at various stages. We would record all of those factors in the subjects of the experiment. Then after completing the experiment, we would have some estimates of the conditional probabilities that are factors in our probabilistic model.

    Now in the decision for the patient, we could either base our recommendation for the patient on the classic RCT, or we could evaluate factors such as age of menarche, age of menopause, age of first pregnancy, family history, availability of digital mammography, interpreter skill to make a recommendation about when to start mammography, how frequently to have it performed, where to have it done, the likelyhood that it will have a benefit, and the likelyhood of a false positive. Since age is not a perfect predictor of breast density, the patient might undergo a baseline exam to determine breast density (then she would also be in a better position to judge the discomfort as well). Then the best decision for that patient can be reached. We would use our more complex model to arrive at the predictions of likelyhood. Furthermore, if there was a dramatic breakthrough in the treatment of breast cancer, we could use the model to change recommendations for screening studies, rather than waiting 13 years for another RCT to be completed.

    Now, in this particular example, the incidence of disease is low in any healthy population except for the elderly. so clinical experience is less likely to lead to functional conditional probabilities, and greater data gathering is necessary for this approach. However, population databases are becoming more complete (an advantage of socialized medicine), and reasonable measure of such probabilities are becoming available.

    Going back to my original suggestion that clinical experience is important in medical decisions. The “good” clinician would have repeated clinical experience with dealing with patients suffering from a disease process. They would have a concept of the disease process (a complex model compared to the simplified model used in an RCT). They would have observation skills (physical exam findings), and interviewing skills (obtaining an accurate history), as well as skills of rapport and empathy. They would assess the factors in their internalized model of the disease process, to make a recommendation for the patient. Their internalized model of the disease process can be updated by published RCT, clinical series, and personal observations.

    In my experience studying medical decision making, we could identify many clinicians using that approach that could beat our methods of discriminant analysis. Depending on how complete the measurements of the factors in the models, and how dependent the models were on the observational skills of the physician, either a small percentage or a large percentage of clinicians could beat the decision from the computer program.

    The argument of what scientific methods that should be applied in different scenarios is not unique to medicine. These are quotes from a lecture published on the internet titled, “Frequentist vs. Bayesian vs. Probabilist” from the quantum optics and biophysics group in the Applied Physics Department at Stanford University. I would assume the author is the professor, Hideo Mabuchi.
    1.However, I think the role of the physicist
    modeling nature is to maximize his winnings in the game of “guess what nature will do”.
    2.The lesson is that we should not trust the statistical inference unless we are convinced that the underlying probabilistic model is realistic.
    Applied physics is much more specific in developing models than medical science. Medical models of disease may become more formalized (in mathematical description) based on successes of advances in medical genetics.

    Now not all clinicians will reach that which I would call a good clinician, it has much to do with problem solving ability and powers of observation (not just academic abilities). Furthermore, no clinician can have universal experience. Finally, clinical load can be overwhelming, and errors can be make in communication of status and results, and doctors can make decisions for nefarious reasons (thanking the pharmaceutical company for the golf trip). Consequently, an approach based on EBM (I won’t call it algorithmic, but that would still be my classification based on medical decision making) can improve practice of medicine. Doctors should be allowed some leeway for clinical experience, because if they are good, then they may have even better results. Whether or not deviations from EBM yield better results can be retrospectively reviewed. If the doctor deviating from EBM has no better results but higher costs, or has worse results, then remedial action can be undertaken.

    So in summary, the best scientific method fro making a decision about an intervention is more like the problem of predicting the weather, rather than determining if there is cause and effect. You wouldn’t base the prediction of weather on Feb 14 from the average of the last ten years. You would base it on weather models and current conditions.

  8. rork says:

    Do the SBM advocates tell us how to obtain a consensus prior on the probability of treatment efficacy, or are there as many versions of SBM as there are practitioners, and if that’s the case aren’t quacks with high priors on their quackery justified (by SBM) in administering it in some cases?
    I’m suggesting some codification might be useful, otherwise it’s just the wild west.

  9. Scott says:

    @rork:

    There’s certainly plenty of room for disagreement on the best way to determine prior probability. But not to the extent of giving acupuncture or homeopathy a significant prior probability. There’s not THAT much wiggle room.

    It’s kind of like how arguing the details of murder vs. self-defense in a case where the dead person broke into the defendant’s house, shot him in the arm, and the defendant shot back and killed him. Sure there’s room for arguing exactly where the proper line lies, but individual cases aren’t necessarily anywhere near the line.

  10. JMB says:

    I would suggest three sources of authoritative priors to stem the wild west approach of “in my experience”.

    1> Probabilistic models based on medical science. Few doctors don’t have some idea in their head about what causes a disease, what the natural course of the disease is, and what may alter the course of the disease. Mostly, it is the physicians in academic positions who have the ability to translate a medical model of disease into a mathematical model suitable for probability calculations. Mathematical models can be tested by prospective studies that are not as involved as RCT’s.

    2> Good clinicians identified by outcome measures become sources for priors. This would not be limited to academic medicine.

    3> Large databases such as those maintained by socialized medicine can become sources. I guess this mixes the frequentist and Bayesian approach.

    These are not mutually exclusive sources of information.

    The probabilistic models will tend to cut down on quackery. The probabilistic models would also be modified by the results of RCT’s and other arguments in the medical literature.

Comments are closed.