Articles

The Role of Anecdotes in Science-Based Medicine

While attending a lecture by a naturopath at my institution I had the opportunity to ask the following question: given the extreme scientific implausibility of homeopathy, and the overall negative clinical evidence, why do you continue to prescribe homeopathic remedies? The answer, as much as my question, exposed a core difference between scientific and sectarian health care providers. She said, “Because I have seen it work in my practice.”

There it is. She and many other practitioners of dubious modalities are compelled by anecdotal experience while I am not.

An anecdote is a story – in the context of medicine it often relates to an individual’s experience with their disease or symptoms and their efforts to treat it. People generally find anecdotes highly compelling, while scientists are deeply suspicious of anecdotes. We are fond of saying that the plural of anecdote is anecdotes, not data. Why is this?

Humans are social storytelling animals – we instinctively learn by the experience of others. My friend ate that plant with the bright red berries and then became very ill – lesson: don’t eat from that plant. This is a type of heuristic, a mental shortcut that humans evolved in order to make quick and mostly accurate judgments about their environment. From an evolutionary point of view it is probably statistically advantageous just to avoid the plant with the red berries rather than conduct blinded experiments to see if it really was the plant that made your friend sick.

Further, the most compelling stories are our own. When we believe we have experienced something directly, it is difficult to impossible to convince us otherwise. It’s just the way humans are hardwired.

Understanding the world through stories was a good strategy in the environment of our evolutionary history but is far too flawed to deal with the complex world we live in today. In fact, the discipline of science developed as a tool to go beyond the efficient but flawed techniques we evolved. Perhaps, for example, your friend became ill because of the raw eggs he consumed earlier in the day, and the plant had nothing to do with it. Evolutionary pressures favored a more simplistic approach to nature, one that tended to assume that apparent patterns were real.

In today’s modern society we are confronted with a dizzying array of apparent patterns and using the simple rules of thumb we evolved to deal with them is not adequate. Whether or not a treatment works for a symptom or disease is a good example. Symptoms tend to vary over time, some may spontaneously remit, and our perceptions of symptoms are susceptible to a host of psychological factors. There are also numerous biological factors that may have an effect. If we are to make reliable decisions about the effects of specific interventions on symptoms and diseases we will need to do better than uncontrolled observation, or anecdotes.

The primary weakness of anecdotes as evidence is that they are not controlled. This opens them up to many hidden variables that could potentially affect the results. We therefore cannot make any reliable assumptions about which variable (for example a specific treatment) was responsible for any apparent improvement.

Here are some specific factors that make it difficult to impossible to reliably interpret anecdotal medical evidence:

Regression to the mean: This is a statistical phenomenon whereby any extreme variation is likely to be followed by a more average variation – by chance alone. Many diseases have variable or fluctuating symptoms – good days and bad days, or periods of exacerbation followed by periods of relative relief. If a person seeks out a treatment when their symptoms are severe, by chance alone this is likely to be followed by a period when the symptoms are not as severe.

Most illnesses are self-limiting: The old saying goes that if you don’t treat a cold it will last for seven days, and if you treat it it will last for a week. Most ailments get better or improve on their own, therefore most treatments will be followed by symptom resolution even if the treatment has no biological effect. More broadly, all illnesses have a natural history, a course they typically follow over time. In order to know if a treatment is affecting that course it has to be compared to patients who are not treated, or receive a different treatment.

Multiple treatments: Often people will try multiple treatments for a disease or ailment making it impossible to tell which treatment had a beneficial effect, if any. Multiple treatments may be taken all at once, or sequentially. For example, a person with a long term illness (but one destined to have a period of relative relief) tries treatment A without effect, then treatment B without effect, then treatment C which is followed by improvements in their symptoms. They then credit treatment C, recounting how multiple other treatments had failed. However, since the person was trying some new treatment most of the time at any point that their symptoms improved there would be a treatment they could credit with that improvement.

Dead men tell no tales (the problem of reporting bias): Cancer survivor groups do not contain people who died of their cancer. Those who die of a disease are not around to give their anecdotes. There is therefore a built in reporting bias. Also, those who feel they were helped by a treatment are much more likely to boast about it than those for whom there was no apparent benefit. People like to tell the tale of the miracle cure they found and had faith in, despite the skeptics and naysayers – but their vision paid off as the treatment worked for them. People have no motivation to recount their experience with the novel treatment that did not work. Further, patients who feel they are being helped by their doctor or practitioner are more likely to return. Those who feel the treatments are not working may not come back at all to report the treatment failure.

Confirmation bias: It is a well-described psychological phenomenon that we tend to seek out and remember information that confirmed what we already believe, or want to believe, and we avoid, forget, or explain away disconfirming evidence.

Vague outcome measures: Good clinical trials use objective outcome measures – those that are binary (like death or survival), quantitative (like a blood level), or are based upon a specific physical finding. Subjective symptoms do not make good outcome measures because they require that judgments be made, and that introduces yet another variable. Should you count those mild sniffles as having a cold? If you are taking a remedy that you think will help you avoid colds you may dismiss those sniffles and report (and even remember) that you did not get any colds while taking the treatment.

The Placebo Effect: The placebo effect is actually a host of many effects that give the appearance of a response to an inactive treatment. These factors include many of the things I listed above, but also other variables that may alter health outcomes or symptoms. See here for a more complete discussion.

The Fallibility of Human Memory: Medical students quickly learn that one of the biggest challenges in taking a medical history is that people are poor historians, which a polite way of saying that human memory is terrible. Anecdotes largely depend upon an individual’s memory of their illness and treatment. This introduces many new variables. There is, for example, a tendency for people to conflate different events in their memory into a single event, or to combine details from various events. There is also a tendency for details to evolve over time to make a story more clean and profound. So people may, in their memory, exaggerate the severity of their symptoms prior to treatment, exaggerate the response to the treatment, clean up the timeline of events so that improvement began very soon after a treatment (rather than before or long after), forget other treatments that were taken, distort what they were told by their various health care providers, etc. I have had countless opportunities to compare a patient’s memory of their illness and treatment to the documented medical records, and the correlation ranges from poor to completely wrong.

For these, and other reasons, scientists have learned not to trust anecdotal reports – or rather to have a realistic assessment of their reliability. This is why it always strikes me as profoundly naive when anyone presents anecdotal evidence as if it is compelling, or even argues that anecdotes should be relied upon as valid evidence.

We also have history to inform our opinions about anecdotes. Western practitioners relied upon the humoral theory of health and illness for thousands of years. Apparently thousands of years of anecdotal experience did not inform them that their treatments were worthless or harmful. Dr. Abrams became wealthy by selling a machine to diagnosis and treat ailments. His devices were widely used, with millions of people swearing by their effectiveness. It worked for them, and their experience was unshakable. When Abrams died it was discovered that his machines (previously protected from inspection) were filled with useless random machine parts. At the turn of the century radioactive tonics were popular, until prominent proponents began seeing the ill effects of radiation poisoning.

The point of these examples is that anecdotal evidence led many people to conclude that these interventions worked. They are useful examples because they are no longer accepted, humoral theory was replaced by scientific medicine, Abrams devices were dramatically exposed, and radiation therapy is directly harmful. But for treatments that are not directly harmful (and least not in an obvious way) or where there is no “man behind the curtain” to dramatically expose, all we have are the anecdotes – and clearly they are not reliable.

Even in mainstream medicine we have learned to distrust anecdotal evidence, even our own. The history of medicine is strewn with treatments that seemed to work but then were abandoned when scientific evidence showed otherwise. The classic example of this is mammary artery bypass for cardiac angina – it seemed anecdotally to work, but it didn’t.

But should anecdotes play any role in medical evidence? Yes, but a very minor and clearly defined one. Anecdotes, with all their weaknesses, are real life experience. It is possible that a treatment does in fact work and personal experience may be the first indication that there is a meaningful biological effect in play. But here are two limiting factors in how anecdotes should be incorporated into medical evidence:

The first is that anecdotes should be documented as carefully as possible. This is a common practice in scientific medicine, where anecdotes are called case reports (when reported individually) or a case series (when a few related anecdotes are reported). Case reports are anecdotal because they are retrospective and not controlled. But it can be helpful to relay a case where all the relevant information is carefully documented – the timeline of events, all treatments that were given, test results, exam findings, etc. This at least locks this information into place and prevents further distortion by memory. It also attempts to document as many confounding variables as possible.

The second criterion for the proper use of anecdotes in scientific medicine is that they should be thought of as preliminary only – as a means of pointing the way to future research. They should never be considered as definitive or compelling by themselves. Any findings or conclusions suggested by anecdotal case reports need to be later verified by controlled prospective clinical studies.

Understanding the nature and role of anecdotes is vital to bridging the gap between the proponents of science-based medicine and believers in dubious or sectarian health practices (as well as the public at large). In my experience it is often the final point of contention between these two camps.

It is interesting to note that the scientific community has long ago made up its collective mind about the weaknesses and role of anecdotes. Logic and the lessons of history speak very clearly on this issue. But there are forces at work today that want to turn back the clock on scientific progress – they want to bring back anecdotes as a reliable source of medical evidence, essentially returning to the pre-scientific era of medicine. In some cases this is done out of frustration – that controlled scientific data has not validated a prior strongly held belief. In other cases it seems to be a calculated attempt to lower the bar of evidence to admit treatments that have not been validated by solid scientific evidence. In either case, this is not in the best interest of the health of the public.

Posted in: Science and Medicine

Leave a Comment (42) ↓

42 thoughts on “The Role of Anecdotes in Science-Based Medicine

  1. sashen says:

    At a recent brunch, I commented that human beings are not very good probabilistic thinkers, at which point a woman defiantly countered, “I am!”

    “Oh?” I said, “How do you know that?”

    “From my own experience!”

    No matter how I tried to explain, she couldn’t understand how it wasn’t possible to use bad probabilistic thinking to determine whether you have good/bad probabilistic thinking.

    Also, I’m reminded of a study where students were given math tests and then received results that were randomly generated — either good or bad. The “bad” students began avoiding math-related classes and projects, now that they “knew” they were bad at math.

    Interestingly, when it was revealed to them that their assessments were totally random… it didn’t change their beliefs or behavior.

    It was only when it was explained to them that humans have a cognitive bias toward holding onto a belief, even if it’s demonstrably false, that they were able to drop their recently acquired self-image as “bad math students.”

    I’ve often seen something similar; a direct attack on the cognitive bias has no effect, but the meta analysis that describes the bias allows people to gain a new perspective.

  2. curt cameron says:

    Thanks for this timely posting. I’ve been thinking for the past couple of days that I should talk to my wife about this exact topic.

    She’s extremely health-conscious – her goal is to live to be 100, and is now 47% of the way there. She voraciously consumes any information she can find about diet and health, and much of the info out there is anecdotal or preliminary. Much of her information comes from Oprah’s guests. A conversation on Sunday showed me that she (my wife) views anecdotes as very persuasive.

    She said that vegan diets are helping people not only avoid getting cancer, but helping to cure the cancers for people who already have it. I don’t know what the science says about this, but I told her it didn’t sound very plausible, and that if the diet had active effects on cancer tissue, you’d have to worry about its side effects and likely wouldn’t be advisable to eat something like that unless you had cancer to cure. She didn’t understand how a “healthy” diet could have side effects – the only side effects would be health, right? And that she hears more and more about people who have beaten cancer with a vegan diet, so it must be true. I didn’t push the issue too much at the time, because these are deeply held beliefs for her, and she thinks my general skepticism is a negative quality. This article should help me bring up the subject (with comments trimmed off, of course!).

  3. Based on a great deal of dealings with alts, I think that the main difference between people who believe in scientific medicine and those who believe in the alternative kind is that the former think that the only way to accurately evaluate the safety and efficacy of drugs and therapies is through objective evidence while the latter believe that the best way to do that is through personal experience going so far as to insist that when good objective studies consistently contradict the conclusions that they have arrived at based on personal experience alone that it is the objective studies that are wrong.

    I personally have no problem with these two different belief systems. What I do have a very big problem with is that almost all the proponents of alt. med. that I have spoken with cannot admit even to themselves that they do not value objective evidence. Most go so far as to believe anecdotes told to them by others without ever even trying to verify them independently.

    One lady once told me dozens of stories of miraculous cures. She got very angry when I kept asking for the contact # of just one of the people she was telling me stories about so that I could see if the person actually believed the tale she told about them. Finally, she said that none of them would agree with her, but of course she was right and they were wrong.

  4. Hypatia says:

    Cue Pec in 5…4…3…2…

  5. rjstan wrote: “I personally have no problem with these two different belief systems.”

    The problem with this position is that it confuses statements of value with statements of fact. It is certainly reasonable to tolerate different belief systems when it comes to aesthetics, culture, and values. However, factual statements are not about belief or opinion.

    The question of whether or not a specific treatment is safe and effective for a specific disease is a matter of fact, not opinion. The question of whether or not anecdotal experience is a reliable measure of safety and efficacy is also a matter of fact, not opinion.

    It is objectively demonstrable that anecdotal evidence is very unreliable when compared to controlled studies.

  6. David Gorski says:

    The problem with this position is that it confuses statements of value with statements of fact.

    Exactly. As they say, you are entitled to your own beliefs, but you are not entitled to your own facts.

  7. apteryx says:

    Yes, any given quantity of anecdotal evidence is far weaker than an equivalent quantity of RCT evidence, but enough human experience added together can produce a large body of knowledge. Most people who deny that our ancestors could ever identify febrifuges by observation try to avoid the issue of how they could identify edible and poisonous plants without double-blind placebo-controlled trials, and thereby the question of how the poor idiots managed to survive to reproduce. You chose to emphasizing the point that plants claimed to be poisonous might not be, which can be true (e.g., poinsettia). Still, if you were stranded in a remote part of the world, and a knowledgeable indigenous person told you that one tree full of fruit was edible while another was deadly poisonous, are you telling me that you would say you had no basis for preferring one over the other?

    Personally, I wish that certain people would stop using anecdotes to claim that black cohosh is hepatotoxic, or ginger and cranberry interact with warfarin, when there are controlled trials that debunk these ideas. I guess everyone tends to embrace the anecdotes that fit their own biases!

    Oh, and a point regarding “vague outcome measures”: This is valid where anecdotes are concerned; a person saying “I took this herb and felt better” is not as interesting as a person saying “I took this herb and my CD4 counts went up.” However, you declare that “good clinical trials” use outcomes based on quantitative measures and physical findings, avoiding questions of the patient’s symptoms and how he feels. This is exactly the sort of attitude that drives people away from the doctor’s office, where all too often people get treatments that improve their “numbers” and worsen their health. People do not come to the doctor for numbers but for symptoms, and if you want to see drug treatments approved and promoted without evidence that the product improves symptoms, you are contributing to your own obsolescence.

  8. Joe says:

    There is an anecdote that I like. It comes from the (US) National Public Radio program “Car Talk.” One host, Ray, had tinnitus of a couple weeks duration, so he went to a specialist. After examining him, the doctor suggested that he wait a bit longer to see if it resolves itself. Ray asked what might cause the problem, and the doctor nonchalantly suggested a brain tumor.

    On the drive home, Ray noticed the ringing in his ears was gone, and it never returned. How about that- cured by nontreatment!

  9. PalMD says:

    @apteryx
    Believe it or not, how patients feel is actually a measurable variable and many studies do address this.

    Also, many studies use an intention to treat model of analysis, which essentially takes into account variability of individuals, but that gets a little complicated.

  10. Roy Niles says:

    As to confirmation bias, you could say it also includes a form of self-persuasion. The process of utilizing one’s memory involves in large part the mind telling itself what it hopes will turn out to be true. This (in my view anyway) allows the “organism” to move forward, at least for the short term. Organisms that take too long to move tend to have a very short reproductive span. Some would say that’s almost the first rule of survival.

    This might then be the basis for an important difference between those who turn to alternative medicine, and those who prefer science. The former clearly place more trust in their short-term calculating apparatus than in our more recently developed long term deliberative “rational” process.

  11. apteryx says:

    PalMD – “Believe it or not,” I am aware that many studies use instruments designed to measure patients’ symptoms and feelings. My point was that Dr. Novella seemed to suggest that good studies should not do so, and thereby that studies whose primary endpoints involve the alleviation of patient discomfort are not good studies.

  12. apteryx,

    I specifically made the point that uncontrolled observations work well enough in our evolutionary environment – addressing such everyday questions as what is safe and good to eat, but they are inadequate to more complex observations. Further, the more immediate and obvious the effect the more accurate casual observation will be. We don’t need controlled trials to tell that a bullet to the back of the had is a bad thing.

    You wrote: “I guess everyone tends to embrace the anecdotes that fit their own biases!”

    This is true, but has absolutely nothing to do with the point that no one should rely upon anecdotes for such questions.

    And to clarify on vague outcome measures – the point is that the more objective the outcome the more reliable the result. Clinical trials try to incorporate at least some objective outcome measures. However, there is also a trend to include quality of life measures. While these are considered “soft” in terms of their quantifiability and reliability, they are thought to enhance the data by giving a bottom line measurement of the net effect of a treatment on quality of life.

    Also – you confused the context of clinical trials, where we want as objective and reliable data as possible, with the context of treating individual patients, where treatments are individualized and subjective outcomes are much more important.

  13. fls says:

    I would also add that people who try alternative treatments may be self-selected to do better (regardless of treatment) than the average person with the disease.

    Those people with more resources (time, education, money, social support) at their disposal tend to have better health and those same people may be more likely to have the opportunity to try these treatments.

    Those people who are less ill also have greater opportunity to try alternative treatments. If you are bedridden, constantly in and out of the hospital, or debilitated by your symptoms, you will find it difficult to make a trip to your local homeopath. And those with rapidly progressive disease will succumb before they can consider trying alternate treatments, selecting out those with more indolent forms for the naturopath.

    Linda

  14. DBonez says:

    I think part of the reason anecdotes are so widely accepted by most or all societies is simply because the very foundation of most people’s core belief system is faith-based. I’ve often called homeopathy and CAM faith-based science since its believed even when no proof exists.

    Using that analogy, it is easy to see why anecdotal evidence is almost universally accepted by people transcending all cultures, income, education, and intelligence levels. Only those with at least a little scientific understanding, skeptics, and those trusting authoritive figures (doctors, scientists, etc.) are probably going to question anecdotal hearsay and seek more evidence. I’m sure of course there are even a number of non-theistic people who will believe faith-based science, but I imagine they are more easily swayed by a good argument.

    With so many industry leaders and policy makers being ardently “faithful,” it’s no wonder science and medicine is infiltrated with anecdotal woo-woo nonsense.

  15. BlazingDragon says:

    Dr. Novella wrote:

    “And to clarify on vague outcome measures – the point is that the more objective the outcome the more reliable the result. Clinical trials try to incorporate at least some objective outcome measures. However, there is also a trend to include quality of life measures. While these are considered “soft” in terms of their quantifiability and reliability, they are thought to enhance the data by giving a bottom line measurement of the net effect of a treatment on quality of life.

    Also – you confused the context of clinical trials, where we want as objective and reliable data as possible, with the context of treating individual patients, where treatments are individualized and subjective outcomes are much more important.”

    I would just add the comment that positive clinical trial benchmarks are absolutely useless unless patients actually “feel” better. I’m glad to hear that subjective outcomes are being added to “hard data” clinical trials. It should be useful for determining which FDA-approved treatment is better for a given patient when the “hard” data are very similar for both treatments.

    As a disclaimer, obviously there are diseases where a “hard” benchmark is the only worthwhile outcome (like cancer going into remission vs. the patient dying). What makes medicine difficult is diseases that are fuzzy to begin with, yet bring significant disability (lupus comes to mind, with all its many measured abnormalities, none of which seem to, individually, correlate well with disease severity and course). Quality-of-life measures for all clinical trials are a welcome addition, but especially so for diseases that seem to defy pigeon-holing.

  16. Steven Novella wrote: “The question of whether or not a specific treatment is safe and effective for a specific disease is a matter of fact, not opinion. The question of whether or not anecdotal experience is a reliable measure of safety and efficacy is also a matter of fact, not opinion.”

    I agree with you. I think the history of civilization clearly demonstrates that and that the vast majority of the public would agree if only they were better educated in history, science, medicine and skepticism. However, I also think that there will always be a small group of people who no matter how much evidence they see to the contrary will continue to believe that personal experience alone is the best or only way to evaluate drugs and therapies and I believe that it is a waste of time to try to convince them otherwise. But we can insist that they be honest with themselves and everyone else. We can insist that they clearly admit that their specific beliefs are not supported by any objective much less scientific evidence. That they are based purely on personal experience. If nothing else, getting them to clearly state that will result in many in their audience placing much less weight on their stories which basically is the best I think we can hope for.

  17. joel_grant says:

    The next time someone tells you “Well it works for me!” you might point out that that is what Tom Cruise says about Scientology.

  18. pmoran says:

    “Understanding the nature and role of anecdotes is vital to bridging the gap between the proponents of science-based medicine and believers in dubious or sectarian health practices (as well as the public at large). In my experience it is often the final point of contention between these two camps.”

    Steve, that is also my experience. I think the medical profession is partly to blame, especially for a somewhat high-handed approach sometimes adopted some decades ago. We even now tend to pronounce against “alternative” anecdotal material via in-house judgments that are a complete mystery to those who find the same evidence compelling. Yet we rarely try to explain precisely why the evidence is deemed weak. You do an excellent job of explaining the general principles involved, but those who most need to understand all this will probably have to be led by the hand through the considerations that most relate to each diverse area of clinical interest.

    We also profess that anecdotal evidence can serve as a guide to matters worthy of further research. Yet we are rarely seen to be responding that way in relation to “alternative” methods. This is another mystery for those who see alternative medicine as offering very promising treatment methods. Just what kind of anecdotal material are we looking for? Do we ever offer guidance? Hence the perception that there is a brick wall of unthinking bias, arrogance and even conspiracy against alternative methods.

    Some skeptics explicitly foster the perception that the only kind of evidence we are responsive to is double blinded placebo controlled studies. These persons are probably confusing the kind of evidence that might be needed for the scientific “proof” of some kinds of clinical claim with that necessary to merely formulate a tenable hypothesis. But they are unwittingly offering promoters of dubious methods a ready-made excuse for being unable to produce ” the kind of evidence the doctors want”.

    My special interest is cancer quackery. It should be easy to support the usual alternative cancer cure claim with quality anecdotal material. Many types of cancer are so predictable that the patient can act as their own control with something like 100,000 to 1 reliability, and cancer is mostly easy to objectively measure with modern technology. Who would not be impressed by the production of even two or three recent cases where advanced, biopsy-proved non-small cell lung cancer regressed with treatment?

    That, I think, is the message that we need to be getting across in relation to dubious cancer cures — not that the anecdotal material is weak, but precisely WHY it is weak.. The onus, and the focus must be thrust back onto those making the claims.

    I offer my own take on the role anecdotal evidence in cancer treatment on my web site e.g. http://www.users.on.net/~pmoran/cancer/Brenneranecdote.htm
    I try to explain what good anecdotal evidence might look like in this particular area of medicine.

    http://www.cancerwatcher.com

  19. Grayson says:

    Excellent post, sir.

  20. pmoran wrote:”We also profess that anecdotal evidence can serve as a guide to matters worthy of further research. Yet we are rarely seen to be responding that way in relation to “alternative” methods.”

    I agree with your comments, but would like to add that the other factor we take into consideration is scientific plausibility or prior probability. Anecdotal experience is generally not sufficient to justify spending limited health care research funds to explore the absurdly implausible.

    Also, part of the reaction against anecdotes supporting dubious modalities is that they are being used to support marketing these modalities with health claims. Context is important.

  21. David Gorski says:

    Indeed it is, as I discussed testimonials a couple of weeks ago.

  22. daedalus2u says:

    Nice post on the actual position anecdotes have in science and medicine. I think I completely share it. My experience is that rather than discounting the evidentiary value of anecdotes to close to zero, many scientists and researchers actually ascribe a negative evidentiary value to it. That is if there is an anecdote about something it is thought to be less likely to be correct that if there are no anecdotes.

    Why some people do this is not something that I understand. I know it doesn’t have to do with being a scientist or a skeptic, I think it is the “once burned twice shy” effect.

    It is really making my efforts to further my research in nitric oxide more difficult. In autism for example I have the anecdotal experience that my Asperger’s got better when I raised my NO level (I noticed the improvement before I realized I had Asperger’s and before I had read up on the physiology of ASDs, so I think a placebo effect is unlikely). It is not just my observation; my psychiatrist (MD from Harvard, PhD from MIT) of more than 15 years noticed these same changes too. I appreciate it is an anecdote.

    Once I started reading up on ASDs, I realized that virtually every physiological symptom associated with ASDs was consistent with physiology being skewed in a low NO direction. The focus of my research is the effect of commensal autotrophic ammonia oxidizing bacteria which live on the skin and metabolize ammonia into NO and nitrite. These bacteria are vey well known in soil chemistry and waste water treatment. They are ubiquitous in the environment. I have found these bacteria living on the surfaces of multiple organisms (vertebrates and invertebrates) living in the wild where they produce NO from endogenous and externally applied ammonia. I have multiple instrumental measures of NO from these bacteria in vivo (human) coincident with an instrumental measure of a physiological effect known to be mediated through NO while the subject was asleep and unable to see the instrumental recordings. I have instrumental measures of NO from these bacteria in vivo (human) due to external application of ammonia. I have instrumental measures showing that some of this NO is absorbed, and that the response time of NO generation and absorption is rapid (less than 1 minute) upon ammonia application (demonstrating plausible physiological linkage between adrenergic sweating and NO generation as a stress compensatory response).

    I keep being told that because my n=1, it is an “anecdote” and of no “scientific” value, or even a negative “scientific” value. What I need is a larger n. As someone with Asperger’s, I don’t have the interpersonal skills to negotiate to get the resources I need to do research with a larger n. When I write up my results and explain them using data and ideas from the literature I am told I am “cherry picking” by people who have never even looked at the NO literature, let alone have read as much as I have (I am not quite hyperlexic). I appreciate that “cherry picking” can be a problem if one does not have a good filter to weed out what is wrong in the literature and apply appropriate reliability factors to what remains, near zero for anecdotes consistent with data, pretty high for data that seems reliable, zero for hypotheses and ideas inconsistent with reliable data. If someone was familiar with the ASD and NO literature they would appreciate that there are essentially no physiological symptoms of ASDs that are not explained by low NO. In my last poster on ASDs (at the Autism Consortium Retreat), I cited 49 papers (for brevity) on NO pathways involved in ASD symptoms (that did not include citations for NO involvement in epigenetic programming). One would have to “cherry pick” to find symptoms not consistent with low NO, and I can’t really think of any.

    It is extremely frustrating to be lumped in with the homeopaths and autism quacks because my “data” is “anecdotal”. I will stop my rant now.

  23. BlazingDragon says:

    pmoran makes a good point, one that should be expanded upon. Dr. Novella wrote in the post that these anecdotes are powerful because they were most likely very useful in the uncomplicated world of hunter-gatherers. I agree that anecdotes are far less useful in today’s complicated world (not to mention that anecdotes have been used to sell untold misery and failed “cures” by quacks of all kinds).

    I would argue though that doctors need to MUCH better explain this to their patients. Doctors often have a hidden hostility (built up by bitter experience with patients who are convinced a quack “cure” works) to anecdotes. But anecdotes are deeply rooted in the human mind, so dismissing them with a scoff will cause an automatic reaction in most people, something along the lines of “I can’t believe the doctor just told me I’m full of crap and that I’m an idiot for even bringing it up.” This leads to a weakening of the doctor-patient relationship.

    I learned a long time ago (while teaching o-chem labs) that what is self-evident to me as a grad student in o-chem (who understands the subject very well) is NOT self-evident to the vast majority of my students. Thus, I need to explain the same simple (to me) concepts over and over again.

    Doctors and skeptics will be able to understand the limited utility of anecdotes without too much trouble, but the vast majority of the population is going to need more than a gruff dismissal of something that makes a lot of sense to them. I understand how frustrating it is to have to explain the same concept several times a day, but it is the only way (short of reforming public schools in a major way) to get the information out on the limited utility of anecdotes in medicine.

    Letting one’s hostility show when trying to explain these simple concepts is very counter-productive (if the goal is to get the patient to understand that their Aunt Matilda’s vodka cure for cancer doesn’t work). It doesn’t matter how well-deserved the hostility toward anecdotes is… This hostility is also usually subconscious, which makes it harder to detect and control.

    I understand all too well the limitations that modern insurances practices place on the time-per-patient, so it is very difficult to actually explain why anecdotes aren’t that useful. But if doctors would, as a rule, dial down the outright hostility and contempt they have for anecdotes, it would be much better for the doctor-patient relationship. It would also suck a lot of the wind out of the sails of quacks who use this hostility to prove that “they were right and the doctor was wrong,” an emotional argument that has a great deal of resonance with the average person. Such emotional arguments are nearly impossible to “reason” a patient out of once they’ve been set in concrete.

  24. Harriet Hall says:

    It may seem like an impossible task to try to explain to true believers why they should mistrust their own experience. But I recently glimpsed a ray of hope. I met a woman who had believed in all kinds of woo-woo things like reflexology, who owned a New Age store of some sort, and who thought she had psychic abilities. Someone told her she could win a million dollars from James Randi for demonstrating her abilities. She firmly believed she could do that. She joined the James Randi Educational Foundation Forum, and over a period of time other forum members explained to her what a valid test would involve, how science is done, and they offered her alternative explanations for her experiences, explanations she had never considered. She has now given up her previous beliefs and is a staunch rationalist. She lost her New Age friends, but has found a whole new community of like-minded skeptics in the JREF.

    It doesn’t happen often, but people CAN change firmly held beliefs. We can’t make them change, but we can offer them the evidence and the opportunity to change.

  25. BlazingDragon says:

    Most of the patients who come into the average doctor’s office aren’t true believers… they just hate having their “intuition” spit back in their face. Doctors facing “true believer” woo idiots develop a gut-level reaction to hearing a person who imitates a “true believer” (kind of like being forced to stick your finger in a light socket repeatedly).

    Most patients are fairly rational, so a simple explanation as to why their “intuition” is wrong (delivered without a look of exasperation or a rolling of eyeballs) will go a long way toward killing this particular beast…

  26. jayh says:

    “She firmly believed she could do that. She joined the James Randi Educational Foundation Forum, and over a period of time other forum members explained to her what a valid test would involve, how science is done, and they offered her alternative explanations for her experiences, explanations she had never considered.”

    I have seen this happen. When we met my wife was a practicing Wiccan (with the crystals), tarot card reader, etc. I never put down her beliefs but did explain how things worked when appropriate. One eye opener was when she commented how powerful crystals were they “powered” watches… I explained the piezoelectric effect and the wonderful temperature stability of quartz, how all this was physics. Her viewpoint changed markedly that day.

    I went through a similar experience as I abandoned Biblical creation beliefs that were taught to me as a child, so I know what she experienced.

    But, now rationalist, she still has a wicked cool collection of tarot cards.

  27. wertys says:

    BlazingDragon is quite accurate with this last observation.

    Many doctors are not able to take a couple of minutes to explain why and how an intelligent lay person could misunderstand things which are obvious to someone who does this for a living. In my field of medical practice, fifteen minutes spent in the first consultation doing just this can save weeks or months of angst down the track, and help frame reasonable, mature and shared goals for what the treatment plan might achieve..

  28. MedsVsTherapy says:

    “It is objectively demonstrable that anecdotal evidence is very unreliable when compared to controlled studies.”

    Unfortunately, RCT cannot be heralded as infallible. Pubmed #10861325: Concato presents measures of effect for a range of conditions where there is evidence (odds ratios) from both observational studies and RCTs; the obs data are more uniform, while the ORs from the RCTs show more variance. Obvious when you consider that the samples for the RCTs are various limited samples drawn from the population that the obs studies sample in a more representative way. Just look at the inclusion/exclusion criteria of any RCT: these incorporate limits directly and indirectly, leading to dispersion of measures of effect.

  29. MedsVsTherapy says:

    I posted a comment abt RCT evidence, but I should have first said: great post about anecdotal evidence.

Comments are closed.