Articles

A Budget of Anecdotes

Anecdotal evidence. An oxymoron? Or a valid approach to understanding data?

The problem is there are different kinds of anecdotes, used for different purposes, but the purpose of anecdotes is rarely if ever defined explicitly. Anecdotes are used for one purpose by one speaker/writer but interpreted in a different context by the listener/reader. People love anecdotes, especially if the anecdotes are about them or their beliefs. Anecdotes are how patients transmit the particulars of their disease to their health care providers. The medical history, as taken from the patient, is an extended anecdote, from which the particulars of the disease have to be extracted. Anecdotes are how physicians explain disease and treatments. Anecdotes are a tool with which teachers instruct their students. Anecdotes are how CAM proponents validate their particular system, and how skeptics invalidate them.

Anecdotes are useful tools for presenting yourself and your ideas. The convention season is over and is was striking how the candidates attempted to win over voters with anecdotes about their lives rather than the particulars of their policies. Using variations of ‘anecdote’ as a pubmed search term yields little of substance. The predominant theme on medline is to contrast anecdotes with evidence, always to the detriment of anecdotes. Anecdotes have power to influence far greater than evidence.

On The Skeptics Guide to the Universe #165 there was an interview with Ben Goldacre, who noted that there was the popular misbelief that the MMR vaccine was a cause of autism. The belief waned not when the voluminous data on the safety and lack of association with autism and the MMR was released, but when it was discovered that the primary proponent of the MMR/autism link received large sums of money to testify about that MMR/autism link. It was the anecdote about his conflict of interest that invalidated the idea, not the science.

The participants of the comments section use and deride anecdotes with equal frequency. I cannot find that a classification of anecdotes has been attempted before, so I am sure what follows is incomplete. I am equally sure the readers of the blog will increase my budget (1). This is a catalogue of anecdotes as they are used in medicine, by clinicians and patients and in discussions of CAM. With, of course, anecdotes.

1) Anecdotes used to describe new observations.

Occasionally an astute clinician notices something new, unexpected, or different in a disease manifestation or a side effect of treatment. If he has the time, it gets published or reported. Otherwise it gets filed away in the back of the brain and next time a similar case is seen, the oddities of the prior case may be remembered and mentioned to others. An anecdote may have a kernel of truth that will be validated later, or be forgotten, or be disproved. I remember as a resident I would comment that if a patient smoked three packs of cigarettes a day, they must be schizophrenic. At the time it was somewhat of a smart ass comment, but later it was shown that smoking helps ameliorate the symptoms of schizophrenia. I need to pay closer attention to my own anecdotes.

Homeopathic remedies are entirely based on anecdotes used to describe new observations, but as practiced on bizarro world. Each remedy has a ‘proving’ (2) where healthy people are given a homeopathic concoction and watched for every symptom, sometimes for days afterwards, which are recorded carefully and used to determine the use of the homeopathic preparation. Each homeopathic proving is an anecdote used as a new observation as to what the homeopathic remedy is to be used for.

2) Anecdotes used as archetype

Diseases, as described in textbooks, can be dry as dust. The medical literature, so called, resembles literature much as the dictionary. The style of medical writing may be maximally efficient for imparting knowledge, but it is also maximally dull. Adjectives and adverbs are frowned upon, and it is the rare article or textbook that has a clever phrase or bon mot. Lists of symptoms and the percentages of their occurrences make up the standard medical description of disease.

Unusual diseases are hard to diagnose if you have never seen it before. Knowing about a disease from reading a textbook does not mean you are likely to recognize it the first time you see it. It is good to have a prior case, an archetype, to which you can compare both the textbook descriptions and the subsequent potential cases.

Take Still’s Disease.

I know the presentation of Still’s: fever, rash, myalgias and arthralgias with sore throat, swollen lymph nodes and hepatosplenomagaly. Not really all that specific in its presentation. This description can also be applied to mononucleosis, some cancers, rheumatic fever and a few other diseases. As mentioned before, the body has a limited number of ways of responding to illness.

The first time I saw a Still’s was in clinic on a Friday afternoon, I had no idea what the patient had. I only knew she looked sick and her labs suggested something very bad was going on. So I admitted her to the hospital and left for the weekend. When I returned on Monday she had been discharged and the hospitalist told me, oh yeah, it was Still’s “like oh yeah, it is raining out”. She was a textbook case, but a textbook never does justice to the patient. Now when I see a fever and I am concerned about Still’s, I think about that particular case and compare her to the case at hand. When I discuss Still’s with the residents or read about Still’s, I use as an archetype that first case as an anchor for understanding.

In medicine, these archetypes are based on the best scientific understanding of disease. When you understand the pathophysiology of Still’s, or Cat Scratch, or any disease and are confronted with a patient who may have a given disease, it is good to have an archetype to which you can refer and apply the pathophysiologic understanding. It puts a face on the disease.

3) Anecdotes used as an example of a general case

This has a long tradition in medicine. I need to give a lecture about Zoonosis (animal related infections). Often the lecture will start with a case presentation. It may be a real case, or it may be a combination of several cases. The purpose of the anecdote is to tie the lecture, which is often filled with the dry results of clinical studies, to a real person with a real disease. Unlike the archetype, which is used to help understand unusual diseases, the anecdote is used to describe common medical conditions and is codified in the case presentation. Anecdotes are effective tools to aid learning about a common disease.

4) Anecdotes used as an example of atypical manifestations of common disease.

We all get blindsided by common diseases that present oddly. Classic angina presents with pain that radiates into the neck and down the left arm, often brought on by exercise. As a resident I had a patient whose angina was left wrist pain with exercise. I like to tell that anecdote to residents to emphasize the need to be on the lookout for atypical manifestations of common diseases. Medicine would be so much simpler if patients would read the textbooks before coming to the hospital.

Corollaries include anecdotes used as examples of typical manifestations of uncommon disease (a cousin of the archetype anecdote) and the anecdote used as examples uncommon manifestations of uncommon diseases. All of these disease manifestations are best remembered through anecdotes, and serve as cautionary tales. If you are not careful, you too may be fooled and miss an important disease.

5) Anecdotes used as an example of proven concepts

Syphilis has a wide variety of manifestations and is best treated with penicillin. Proven concepts. When I teach a resident or discuss a case, I may use an anecdote about one of the many syphilitics I have treated over the years.

Homeopathy cannot work for any disease except thirst, since all homeopathic concoctions contain nothing but water. A proven concept in so far as it is possible to prove a negative. I could present a case where a patient used a homeopathic remedy for syphilis and the disease progressed unhindered by the homeopathic concoction.

Both the efficacy of penicillin for the treatment of syphilis and the lack of efficacy of homeopathy are proven concepts, illustrated by anecdotes.

6) Anecdotes used as an example to prove an unproven concept

This would be an example of the justly derided anecdotal evidence, and differs from the other, perhaps more benign, forms of anecdotes.

Homeopathy does work, here is the proof: I gave a patient with joint pain a homeopathic concoction and the patient improved. The knee pain went away. Proof of the efficacy is in the successful intervention.

It is here that the motto “the plural of anecdote is anecdotes not data” is most applicable. It is whether the anecdote is being used to prove a concept, or used as an example of a proven concept is where practitioners of CAM and real medicine differ, both on how the anecdote should be used and the validity of the anecdotes use.

It is the use of anecdotes as a proof of concept upon which the basis of much of CAM rests, and is the most unreliable use of anecdotes. Given the vagaries of human memory, the difficulty in proving causality, and the lack of biologic and physical possibility of most CAM interventions, using anecdotes as proof of CAM effectiveness is a particularly problematic use of anecdotes. Even if Dr. Weil tarts them up by calling them uncontrolled clinical observations.

Saying “the plural of an uncontrolled clinical observation is uncontrolled clinical observations, not data” just doesn’t seem as catchy.

It is this kind of anecdote that has the most power, both in CAM and in the reality-based practice of medicine. An anecdote where something happened has far more power to convince than an anecdote where nothing happens. A patient has an adverse reaction after a vaccine or gets better after taking echinacea. The anecdote suggests causality. If you respond that clinical studies demonstrate the lack of data to support their anecdote, they will look at you like a dorky brainiac divorced from the real world. If you counter with an anecdote where, under the same circumstances, for you nothing happened after the vaccine or the echinacea, you will probably get the same “man what a clueless dork” roll of the eyes. Hits have far more power than misses, your anecdote will be dismissed, and you will be branded a hypocrite since in the past you have called anecdotes worthless.

Skeptics of CAM are doomed to forever lose against the formidable power of the positive anecdote, as one positive anecdote is worth 1000 studies and 10,000 negative anecdotes.

Can anecdotes ever be considered proof? There may be times when anecdotes can be definitive without the need for classical clinical trials (3). Some types of adverse drug reactions may be considered proven with anecdotes. In “Anecdotes that provide definitive evidence” the authors provide 4 instances that they feel anecdotes are sufficient data to prove a hitherto unproven idea.

1) Extracellular or intracellular tissue deposition of the drug or a metabolite
2) Specific anatomical location or pattern of injury
3) Physiological dysfunction or direct tissue damage that can be proved by physicochemical testing
4) Infection as a result of administration of a potentially infective agent or because of demonstrable contamination.

These are all very narrow instances of anecdotes as proof, and may be unreliable, as case reports for adverse drug reactions are rarely confirmed in subsequent studies (4).

I would consider a stroke immediately following cervical spine manipulation in the same category where the anecdote is convincing. I am sure others will disagree or have other anecdotes they consider definitive. What level of plausibility where an anecdote could be considered sufficient proof to confirm a hypothesis is uncertain, depending on prior plausibility and will vary from person to person.

But as proof of a concept in CAM, in most instances one has to combine aphorisms: “the plural of anecdote is anecdotes, not data” and “extraordinary claims require extraordinary evidence.”

6) Anecdotes used to guide treatment when there is no data.

Grand Rounds this week was on Pulmonary Hypertension, a rare disease with an array of new, effective, and very expensive therapies. Pulmonary hypertension has a variety of causes and stages. When do you start therapies and in what order. There is some data, but given the rarity of the disease, there are many instances where the particular case does not match the known information. So as a clinician you rely on the anecdotes of others and your past experience to guide therapy.

The same issue occurs in AIDS treatment. There are 28 different HIV medications and more coming. There is some data to suggest the best initial therapy, but as resistance and intolerance occurs to the medications, often it is anecdotal data that determines which medications are used.

Clinicians get comfortable with one therapy and perhaps had poor outcomes with other therapies, and these anecdotes influence further interventions. Unlike CAM, there is biologic plausibility behind the anecdotes for your interventions. You always understand that using anecdotes to help decision making is a temporizing measure because, and again unlike CAM, there will be future studies that will help guide optimal therapy.

7) Anecdotes used as the history of the present illness

People love stories. They love to tell stories, especially about themselves and about their diseases. Patients do not give organized medical histories. They tell extended anecdotes about their illness. What a patient tells me about their disease and what really happened are often two different issues and all too often (especially when it is the last consult on a Friday) the patient is particularly loquacious with the details of their disease that have no bearing of the diagnosis. Part of being a doctor is learning to extract the pertinent information from the sometimes endless detail in the anecdotes.

8) Anecdotes used to provide understanding of the unknown.

Often the diagnosis and treatment of a disease can be uncertain, and by referring to themselves and others with anecdotes about the process, it puts the illness in a context that renders it explainable to the patient and their close friends and family. Patients frequently ask my experience with their disease, and I used anecdotes about past successes and potential pitfalls to explain and personalize their disease. The anecdotal experiences are, I think, more reassuring and understandable, than just recitation of the statistics of their disease, although I provide that as well. Patients don’t care that 98% of people with streptococcal endocarditis are cure, since for them it is all or nothing. So I mention the success rate, the outcomes, and allude to the experiences I have had with other patients.

9) Anecdotes used to guide diagnosis or therapy because of either prior unusual outcomes, good or bad.

This is the hard one, as a clinician, to be conscious of. If you have ever had a particularly horrible outcome from the care of a patient, you will remember that case. Vividly and forever. When you have a similar case, that anecdote will float though your head and may alter your approach, perhaps inappropriately so. Bad outcomes and complications have enormously greater impact on care than good outcomes, and I bet most of us made rounds with the ghosts of patients past, who look over our shoulders and guide our therapy. These anecdotal ghosts can exert more power than evidence, especially when the data is sparse or the outcome was particularly bad.

It is rare for anecdotes to be used in a single context. Language is not precise and anecdotes can have many interpretations. Anecdotes are flexible and one anecdote can have many functions to the user and many interpretations by the listener. Perhaps we should be more precise in our use of anecdotes. When professional societies make recommendations, the strength of the data has a rating system, of which there are a variety (5).

Perhaps, and I am only half serious when I suggest this, we need a similar classification for anecdotes when we discuss medicine and, especially, CAM.

Someone tells an anecdote about how CAM works or how it fails. How is it being used? After the anecdote, put the type of anecdote in a parenthesis, just like they big boys do when making recommendations. Then we can assess the validity of the anecdote in the context the writer intended, rather than the context applied by the reader.

Anecdotal Types

Type 1) Anecdotes used to describe new observations.
Type 2) Anecdotes used as archetype.
Type 3) Anecdotes used as an example of a general case.
Type 4a) Anecdotes used as an example of atypical manifestations of common disease.
Type 4b) Anecdotes used as examples of typical manifestations of uncommon disease.
Type 4c) Anecdote used as examples uncommon manifestations of uncommon diseases.
Type 5) Anecdotes used as an example of a proven concepts.
Type 6) Anecdotes used as an example to prove an unproven concept
Type 7) Anecdotes used as the history of the present illness.
Type 8) Anecdotes used to provide understanding of the unknown.
Type 9a) Anecdotes used to guide diagnosis or therapy because of either prior unusual outcomes, good.
Type 9b) Anecdotes used to guide diagnosis or therapy because of either prior unusual outcomes, bad.

I have an anecdote that suggests such an approach will work (Anecdote Type 1).
—————————
References etc

1) Budget: archaic a quantity of material, typically that which is written or printed. An old classic is a Budget of Paradoxes.

2) Like complementary, which complements nothing, and alternative, which isn’t, a ‘proving’ proves nothing.

3) Anecdotes that provide definitive evidence 2006;333;1267-1269 BMJ . Jeffrey K Aronson and Manfred Hauben

4) BMJ. 2006 Feb 11;332(7537):335-9. Case reports of suspected adverse drug reactions–systematic literature survey of follow-up.

5) BMC Med Res Methodol. 2006; 6: 52. A system for rating the stability and strength of medical evidence

Posted in: Science and Medicine

Leave a Comment (9) ↓

9 thoughts on “A Budget of Anecdotes

  1. Karl Withakay says:

    “Hits have far more power than misses.”

    “Skeptics of CAM are doomed to forever lose against the formidable power of the positive anecdote, as one positive anecdote is worth 1000 studies and 10,000 negative anecdotes.”

    For me, these statements are the cream of an excellent post. This is a huge reason why people buy into woo.

    This is why cold readers are perceived as genuine psychics.

    This is why one poorly done, unblinded, uncontrolled study supporting the efficacy of acupuncture for controlling hot flashes holds infinitely more weight to CAM supporters than all the numerous well done, controlled, double blinded RCTs that fail to demonstrate that acupuncture significantly outperforms placebo for anything.

  2. overshoot says:

    A useful phrase in my field (engineering) is “extrapolation from a point.”

    Since most of us were required to load up on maths, this one can be relied upon to produce the desired cognitive dissonance.

  3. daedalus2u says:

    In the paper you cited, “Anecdotes that provide definitive evidence”, I find their arguments not persuasive. Revere very recently posted an excerpt from a book on statistics that illustrates that finding a thief in the act, is not the definitive proof of guilt the authors suggest.

    http://scienceblogs.com/effectmeasure/2008/09/bpa_causation_and_scientific_r.php#more

    I always think of anecdotes (if they are honest and not fraudulent) as data of extremely limited statistical significance. That is not how most people see them.

    My experience is that the definition of anecdote is somewhat loose and means what ever detractors want it to mean. In many (if not most) cases the presence of an anecdote makes all other data and arguments less persuasive. In the jargon of Dr. Atwood, the perceived prior probability of the combination of literature data plus a theoretical rational is reduced if an anecdote is added. Anecdotes are treated as data of negative significance. In other words, if you have an anecdote, your hypothesis must be wrong.

    I have been trying to get people interested in the nitric oxide producing commensal bacteria I am working with as being important in human physiology. I have instrumental measures of NO production by the bacteria (in vivo, human), coincident with instrumental measure of a physiological event known to be mediated by NO (in vivo, human), on multiple instances but on only a single subject. I have instrumental measures showing that the bacteria produce NO in vivo, and that some of that NO is absorbed in vivo.

    I don’t dispute that there are many anecdotes showing that anecdotes are in general unreliable. I don’t understand why a positive anecdote is given negative weight.

  4. w_nightshade says:

    I don’t dispute that there are many anecdotes showing that anecdotes are in general unreliable. I don’t understand why a positive anecdote is given negative weight.

    I could be wrong, but I thought that was a pillar of Dr. Crislip’s argument – not all anecdotes SHOULD be given negative weight – they should be evaluated on their purpose and afforded the appropriate level of respect based on that purpose.

  5. Flex says:

    Excellent essay, and it gave me great pleasure to see a referance to Augustus De Morgan’s classic budget.

    Cheers!

  6. daedalus2u says:

    nightshade, you are missing my point. No anecdote or any other piece of positive data should ever be used to give negative weight to an argument. Even a false statement shouldn’t be used to give negative weight to an argument. At worst it provides zero weight.

    A statement that has nothing to do with the argument and has no bearing on it adds zero weight to the argument. It doesn’t take away from other aspects of the argument.

    That people do use the presence of an anecdote to disbelieve what the anecdote is about isn’t about logic and reason, it is about what drives people’s beliefs other than logic and reason.

  7. Joe says:

    There is an anecdote I like. NPR (USA radio) has a show called Car Talk. One host (Ray) developed tinnitus that eventually interfered with sleeping. He went to a specialist who, after several tests, recommended waiting a bit longer before doing anything more drastic.

    Ray asked what it might be, and the doctor nonchalantly relied “It could be a brain tumor.” Ray was shocked, nevertheless he drove home. During that ride, he realized the tinnitus was gone; and, years later, it has not returned.

    When someone tells me a quack cured a long-standing problem, I recall that a cure can (sometimes) be achieved with no treatment at all.

  8. daedalus2u says:

    How I see anecdotes, an analogy to a bridge.

    I see the goal of treatments in medicine as similar to the goal of a bridge, a path to go from a region of ill health to a region of better health. It is vastly more complicated because what is being “bridged” is not understood. At one extreme, one can treat the “bridge” and each endpoint as a “black box” the inner workings of which are completely unknown. At the other extreme, one can know and understand each end point and the physics of the bridge in very great detail.

    With a complete absence of knowledge of the details, a successful anecdote of treatment is analogous to a successful passage over the bridge while wearing a blindfold. You made it over, but you may have just missed falling through a hole in the bridge. That is the analogy of a successful anecdote in CAM. A successful treatment while wearing the blindfold of ignorance. The treatment may have been successful, but you also might have stayed in the same place you were before. You can’t tell because you were wearing a blindfold.

    If there were enough passages to get statistics, one could start to figure out how many holes there are in the bridge by how many don’t make it across. You need to have the number that tried, as well as the number that succeeded. Looking only at successful crossings tells you nothing because 100x more could have tried and not made it.

    Evidence based medicine looks at the number that tried and the number that succeeded and tries to figure out how good the bridge is. You can do that while still wearing blindfolds but you need lots of bridges and lots of passages over it.

    Science based medicine takes off the blindfold and tries to look at and understand the details of bridge construction and of passage on the bridge. Are there holes? If so, change the path to go around them. What is the bridge made of? What loads can it support? Is the factor of safety 10x or 1.05x? What is the coefficient of friction of the roadbed, can slipping be prevented? Is there a railing, can falling off the edge be prevented? Does the bridge vibrate? Is the time on the bridge short enough that supplemental food and water isn’t required for crossing? Can the passage be accelerated by using a vehicle? Are their episodic high winds? Do you need to duck to avoid hitting your head? Are there better bridges?

    A successful crossing by the path laid out by understanding some of the details of the bridge construction is not equivalent to a successful crossing while wearing a blindfold. If you understand enough of the details, you can produce a successful crossing over a bridge that would be impossible to pass while blindfolded. An analogy here would be the successful treatment of a specific type of cancer with a specific type of chemotherapy where “random” chemotherapy to treat generic “cancer” would be unsuccessful.

    CAM proponents can’t tell the difference because they are wearing the blindfold of ignorance and are unable or unwilling to take it off. Until you become expert in the physics of bridge design you are wearing a blindfold. Virtually everyone who wants to cross the bridge is ignorant of bridge design. They have to evaluate the expertise of those who built and are controlling the bridge (and charging the toll) to figure out if paying the toll and trying to cross makes sense.

  9. KarlS says:

    Excellent essay.

    Proposing from the lyrics of Simon And Garfunkel:

    10) Anecdotes used to provide hope (in dire situations). Likely the appeal of CAM among patients with cancers.

    ” … Ive squandered my resistance for a pocketful of mumbles, such are promises. All lies and jest, still the man hears what he wants to hear And disregards the rest, … ” ~ The Boxer

    Many thanks. ~ Karl

Comments are closed.