This is an addendum to my previous entry on Bayesian statistics for clinical research.† After that posting, a few comments made it clear that I needed to add some words about estimating prior probabilities of therapeutic hypotheses. This is a huge topic that I will discuss briefly. In that, happily, I am abetted by my own ignorance. Thus I apologize in advance for simplistic or incomplete explanations. Also, when I mention misconceptions about either Bayesian or “frequentist” statistics, I am not doing so with particular readers in mind, even if certain comments may have triggered my thinking. I am quite willing to give readers credit for more insight into these issues than might be apparent from my own comments, which reflect common, initial difficulties in digesting the differences between the two inferential approaches. Those include my own difficulties, after years of assuming that the “frequentist” approach was both comprehensive and rational—while I had only a cursory understanding of it. That, I imagine, placed me well within two standard deviations of the mean level of statistical knowledge held by physicians in general.
Call me naive, but I did not expect the volume or the emotional depth of the responses to the Iraqi civilian death post. I thought many would respond to the new NEJMed survey as I did; wondering about the validity of the previous surveys and recognizing that they have a validity problem. And, that there is a question about what is printed in major journals, from unexpected sources. I did not mean that studies such as Lancet II not be printed. I stated that it should not have been printed in a first line journal for the general medical public. It could have been printed in a 2nd or 3rd line specialty journal where its methods and conclusions could have been debated and reforms shaped by colleagues. I find that hints and clues to errors in pseudoscientific reports mostly lie in the methods section. But questioning a study’s validity can involve more than just a knowledge of the methods and recalculation of the data. Because the “CAM” movement has redefined the borders of the playing field as well as the rules of the game, the entire environment of the scientific system surrounding implausible or unusual reports has to be examined – this goes beyond limits of methods, and includes motivations, funding, characters, and subtexts.
In developing criteria for estimating plausibility (prior probability) the most important criterion of course is consistency and consilience with established knowledge. But there are more. One can increase the effectiveness of investigation by using indicators not presently included in “Evidence Based Medicine” or in science, but that are used in criminology (previous arrests, convictions,) business (trustworthiness, profit vs loss,) and ideology and politics (elevation of the trivial, manipulation of the system; example: sectarian medicine.)
The Food and Drug Administration (FDA) is proposing a very interesting loosening of their regulations of pharmaceutical company marketing. The pros and cons of the proposed changes present an interesting dilemma, with legitimate points on both sides.
When the FDA approves a drug it is approved for a very specific medical indication. I have long thought that FDA approved indications for drugs were too narrow and restricting. For example, most anti-seizure medications are initially approved not for seizures but only for certain types of seizures – for example for adjunctive therapy (meaning it is meant to be added to another drug rather than used alone) for focal onset seizures (and not against primary generalized seizures – or ones that begin all over the brain at once).
Once approved physicians are free to use drugs as they see fit. If evidence shows that a new seizure medication is effective as first line treatment, then it is ethical good medicine to use it that way, even if it is not FDA approved for that use (this is called off-label use). FDA approved is not equivalent to science-based.
You’re a patient. That cold just isn’t getting better and you have purulent drainage from your nose, and your face hurts and your teeth hurt. You probably have sinusitis, right? You go to a doctor to get an antibiotic.
You’re a doctor. Deep down, you know there’s a good chance the patient has a self-resolving condition. You’d rather not do x-rays on every patient who presents with these symptoms, because x-rays are expensive, expose the patient to harmful radiation, and they are not always accurate. You could puncture the sinuses and take a sample for bacteriological culture, but that’s expensive, painful, and the patient would NOT appreciate it. The patient may not really need treatment, but you want very much to do something to help. If you can find a reason to give the patient an antibiotic, you can feel that you have done something worthwhile. Antibiotics don’t work for a viral infection, but you rationalize that you’re not 100% sure it’s not bacterial, and that sometimes a bacterial infection develops superimposed on a viral infection and mild bacterial infections can develop into severe ones with complications, and maybe you could ward that off. You convince yourself that it really would be prudent to prescribe an antibiotic. Both you and your patient are happy. The patient gets better. You remember this pleasant experience and are reinforced to do the same next time.
Maybe that’s not such a good idea.
A recent study in JAMA showed that antibiotics were not superior to placebo for treating bacterial sinusitis diagnosed by the clinical criteria used by many primary care physicians. There is a growing concern that we have been overtreating sinus infections. Recent research has also shown that we were overtreating ear infections in children, that many of them resolve just as fast without treatment. That doesn’t mean antibiotics should never be used for ear infections. There are now guidelines for using age and clinical presentation to determine which children to treat and which ones can be safely observed without antibiotics. Most of these observed ear infections will resolve but some will eventually require antibiotics too. The situation with sinusitis is similar: most patients may not need antibiotics, and we’re trying to thrash out better criteria for identifying those who will.
Critics point to this kind of thing as a defect of conventional medicine. We use treatments that are not based on good evidence. We do things for years and then find out we were wrong. New studies are constantly contradicting older studies. We keep changing our minds. (more…)
Ever since there have been vaccines, there has been an antivaccination movement. It began shortly after Edward Jenner discovered how to use the weaker cowpox virus to induce long-lasting immunity to smallpox, there has been resistance to the concept of vaccination, a resistance that continues to this very day. Reasons for this resistance have ranged from religious, to fear of injecting foreign substances, to simple resistance to the government telling people what to do. Some fear even the infitessimally small risk that vaccines pose for the benefit of resistance to disease far more than they fear the diseases themselves, a result of the very success of modern vaccines. Of course, vaccines, like any other medical intervention, are not without risks, making it easy for them to jump on any hint of harm done by vaccines, whether real or imagined, even though vaccines are among the very safest of treatments.
One of the biggest myths that antivaccinationists believe and like to use to stoke the fear of vaccines is the concept that they are full of “toxins.” The myth that mercury in the thimerosal preservative commonly used in vaccines in the U.S. until early 2002 was a major cause of autism is simply the most recent bogeyman used to try to argue that vaccines do more harm than good, as was the scare campaign engineered in response to Andrew Wakefield’s poor science claiming a link between the MMR vaccine and autism. Now that study after study have failed to find or corroborate a link between thimerosal in vaccines or vaccines in general and autism to the point where even the most zealous of zealots are having a hard time defending the claim that mercury in vaccines cause autism any more, predictably the campaign against vaccines has fallen back on the old “toxins” myth. If you peruse antivaccinationist websites, it won’t take long to find articles claiming that vaccines are full of the most terrifying and nasty toxins. Examples in the media abound as well. For example, Jenny McCarthy, comic actress and former Playboy Playmate who has been doing the talk show and publicity circuit lately to plug her book in which she claims that vaccines caused her son’s autism and that she was able to cure it with “biomedical” interventions and diet, recently gave an interview in which she said:
What I really am is “anti-toxins” in the vaccines. I do believe that there is a correlation between vaccinations and autism. I don’t think it’s the sole cause, but I think they’re triggering–it’s triggering–autism in these kids. A really great example is…is, sometimes obesity can trigger diabetes. I do believe that vaccines can trigger autism…It’s so much more than just mercury. That is one ingredient in the recipe of autism…I’m talking about all of them. I’m calling for cleaning out the toxins. People don’t realize that there is aluminum, ether, antifreeze, still mercury, in the shots…People are afraid of secondhand smoke, but they’re OK with injecting the second worst neurotoxin on the planet in newborns.
Another example of what I sometimes call the “toxin gambit” comes from Deirdre Imus, wife of shock jock Don Imus, with both husband and wife being well-known and reliable media boosters of the claim that vaccines somehow cause autism:
So, where are the evidenced based (conflict free) studies that prove the safety of these “trace” amounts and proof that there are “no biological effects” of any amount of mercury being injected into our children and pregnant moms? Also, where are the evidence based studies proving the safety of vaccines given to pregnant moms and our children that contain other toxins such as aluminum and formaldehyde?
The most recent example of this tactic comes from an organization called Generation Rescue, which just last week ran a full-page ad in USA Today, paid for in part by Jenny McCarthy and her present boyfriend Jim Carrey:
Besides being one of the most egregious examples of a post hoc ergo propter hoc fallacy that I’ve ever seen from an antivaccination site, this Generation Rescue ad demonstrates clearly a new strategy (or, more properly, a resurrection of an old technique) now that science is coming down conclusively against mercury in vaccines as a cause of autism, a strategy of propagating fear by linking vaccines with “toxins.” So what’s the real story? Are there really deadly toxins in vaccines that parents should be worried about?
This is actually the second entry in this series;† the first was Part V of the Homeopathy and Evidence-Based Medicine series, which began the discussion of why Evidence-Based Medicine (EBM) is not up to the task of evaluating highly implausible claims. That discussion made the point that EBM favors equivocal clinical trial data over basic science, even if the latter is both firmly established and refutes the clinical claim. It suggested that this failure in calculus is not an indictment of EBM’s originators, but rather was an understandable lapse on their part: it never occurred to them, even as recently as 1990, that EBM would soon be asked to judge contests pitting low powered, bias-prone clinical investigations and reviews against facts of nature elucidated by voluminous and rigorous experimentation. Thus although EBM correctly recognizes that basic science is an insufficient basis for determining the safety and effectiveness of a new medical treatment, it overlooks its necessary place in that exercise.
This entry develops the argument in a more formal way. In so doing it advocates a solution to the problem that has been offered by several others, but so far without real success: the adoption of Bayesian inference for evaluating clinical trial data.
When I think back to my own ‘discovery’ of the skeptical movement, it grew out of my experience watching the James Randi Secrets of the Psychics NOVA special. After being enthralled with the special (and with several Randi books already in my library) I sought Mr. Randi out on the Internet. In chat rooms, blogs, forums and skeptical conferences such as TAM this is a tale I’ve heard repeated many times; folks heard about the JREF of CSICOP (now CSI) and then used the World Wide Web to learn more about these organizations.
Recently I began to wonder about my own personal pet peeve (unscientific medicine) and how it has benefited from the Web’s huge explosion and influence. Certainly there are plenty of great sites out there that help to show much of so-called Alternative Medicine for what it really is – blogs like this, Dr. Stephen Barrett’s Quackwatch.org site, the National Council Against Health Fraud, and many other important sites; still the number of sites extolling the virtues of science and critical thinking pale in comparison to those that forward notions embracing magical thinking and quack-related products and health claims. A quick examination of the web’s most popular search tool (Google) shows us the cold hard facts about who’s winning the war of medical woo:
The core principle of science-based medicine is that health care decisions should be based upon our best current scientific evidence and understanding. When applied to the regulation of health products this means that health claims should first be required to meet some reasonable threshold of scientific evidence before they are allowed. Admittedly this is not a purely scientific question but the application of scientific knowledge to an essentially political question – the balance of protection vs freedom.
Regardless of where one thinks this balance should be, I think most would agree that is it a problem if the public generally wants more protection than it is getting, or believes it is currently getting more protection than it is. A Harris poll from 2002 indicates that the majority of Americans believe that companies cannot make health claims about supplements unless they have been proven scientifically and approved by the FDA, when in fact this is not the case. The Dietary Supplement Health and Education Act of 1994 (DSHEA), largely through the efforts of Senator Orin Hatch from Utah, removed supplements from the control of the FDA and specifically allowed for so-called structure function claims to be made about products without any burden of proof. Most Americans are not aware of this fact.
Patients with heartburn are often diagnosed with GERD (gastroesophageal reflux disease) and treated with a drug called a proton pump inhibitor (PPI) to reduce stomach acid production. It is pretty effective, but it doesn’t always work. When it doesn’t, standard practice has been to double the dose of PPI. Doubling the dose only improves symptoms in 20-25%. Most patients who fail the single dose turn out to have normal esophageal acid exposure, or “functional” heartburn. In other words, the symptoms appear to be due to something other than excess acid – so it really may not make much sense to double the PPI dose. What else could doctors try?
Hype over science: Does acupuncture really improve the chances of success for in vitro fertilization?
There it was on Friday greeting me on the ABC News website: “Study: Acupuncture May Boost Pregnancy” in bold blue letters, with the title of the webpage being “Needles Help You Become Pregnant.” The story began:
It sounds far-fetched sticking needles in women to help them become pregnant but a scientific review suggests that acupuncture might improve the odds of conceiving if done right before or after embryos are placed in the womb.
The surprising finding is far from proven, and there are only theories for how and why acupuncture might work. However, some fertility specialists say they are hopeful that this relatively inexpensive and simple treatment might ultimately prove to be a useful add-on to traditional methods.
By the end of the day, the story was all over the media, including radio, TV, news websites, the blogosphere, and various other outlets, all trumpeting the message that a scientific study says that acupuncture can help infertile couples conceive. Nary a skeptical word seemed to be found. Knowing very well just how far parents will go to conceive, I was curious: Did this study actually say what the media says it said? What was so new and radical about this study that it rated a press release and a lot of promotion? Do we here at SBM (particularly Steve) need to rethink our extreme skepticism about acupuncture, given the poor quality evidence and lack of even a glimmer of a convincing physiologic mechanism to explain its supposed activities?