Sep 25 2008
Anecdotal evidence. An oxymoron? Or a valid approach to understanding data?
The problem is there are different kinds of anecdotes, used for different purposes, but the purpose of anecdotes is rarely if ever defined explicitly. Anecdotes are used for one purpose by one speaker/writer but interpreted in a different context by the listener/reader. People love anecdotes, especially if the anecdotes are about them or their beliefs. Anecdotes are how patients transmit the particulars of their disease to their health care providers. The medical history, as taken from the patient, is an extended anecdote, from which the particulars of the disease have to be extracted. Anecdotes are how physicians explain disease and treatments. Anecdotes are a tool with which teachers instruct their students. Anecdotes are how CAM proponents validate their particular system, and how skeptics invalidate them.
Anecdotes are useful tools for presenting yourself and your ideas. The convention season is over and is was striking how the candidates attempted to win over voters with anecdotes about their lives rather than the particulars of their policies. Using variations of ‘anecdote’ as a pubmed search term yields little of substance. The predominant theme on medline is to contrast anecdotes with evidence, always to the detriment of anecdotes. Anecdotes have power to influence far greater than evidence.
On The Skeptics Guide to the Universe #165 there was an interview with Ben Goldacre, who noted that there was the popular misbelief that the MMR vaccine was a cause of autism. The belief waned not when the voluminous data on the safety and lack of association with autism and the MMR was released, but when it was discovered that the primary proponent of the MMR/autism link received large sums of money to testify about that MMR/autism link. It was the anecdote about his conflict of interest that invalidated the idea, not the science.
The participants of the comments section use and deride anecdotes with equal frequency. I cannot find that a classification of anecdotes has been attempted before, so I am sure what follows is incomplete. I am equally sure the readers of the blog will increase my budget (1). This is a catalogue of anecdotes as they are used in medicine, by clinicians and patients and in discussions of CAM. With, of course, anecdotes.
1) Anecdotes used to describe new observations.
Occasionally an astute clinician notices something new, unexpected, or different in a disease manifestation or a side effect of treatment. If he has the time, it gets published or reported. Otherwise it gets filed away in the back of the brain and next time a similar case is seen, the oddities of the prior case may be remembered and mentioned to others. An anecdote may have a kernel of truth that will be validated later, or be forgotten, or be disproved. I remember as a resident I would comment that if a patient smoked three packs of cigarettes a day, they must be schizophrenic. At the time it was somewhat of a smart ass comment, but later it was shown that smoking helps ameliorate the symptoms of schizophrenia. I need to pay closer attention to my own anecdotes.
Homeopathic remedies are entirely based on anecdotes used to describe new observations, but as practiced on bizarro world. Each remedy has a ‘proving’ (2) where healthy people are given a homeopathic concoction and watched for every symptom, sometimes for days afterwards, which are recorded carefully and used to determine the use of the homeopathic preparation. Each homeopathic proving is an anecdote used as a new observation as to what the homeopathic remedy is to be used for.
2) Anecdotes used as archetype
Diseases, as described in textbooks, can be dry as dust. The medical literature, so called, resembles literature much as the dictionary. The style of medical writing may be maximally efficient for imparting knowledge, but it is also maximally dull. Adjectives and adverbs are frowned upon, and it is the rare article or textbook that has a clever phrase or bon mot. Lists of symptoms and the percentages of their occurrences make up the standard medical description of disease.
Unusual diseases are hard to diagnose if you have never seen it before. Knowing about a disease from reading a textbook does not mean you are likely to recognize it the first time you see it. It is good to have a prior case, an archetype, to which you can compare both the textbook descriptions and the subsequent potential cases.
Take Still’s Disease.
I know the presentation of Still’s: fever, rash, myalgias and arthralgias with sore throat, swollen lymph nodes and hepatosplenomagaly. Not really all that specific in its presentation. This description can also be applied to mononucleosis, some cancers, rheumatic fever and a few other diseases. As mentioned before, the body has a limited number of ways of responding to illness.
The first time I saw a Still’s was in clinic on a Friday afternoon, I had no idea what the patient had. I only knew she looked sick and her labs suggested something very bad was going on. So I admitted her to the hospital and left for the weekend. When I returned on Monday she had been discharged and the hospitalist told me, oh yeah, it was Still’s “like oh yeah, it is raining out”. She was a textbook case, but a textbook never does justice to the patient. Now when I see a fever and I am concerned about Still’s, I think about that particular case and compare her to the case at hand. When I discuss Still’s with the residents or read about Still’s, I use as an archetype that first case as an anchor for understanding.
In medicine, these archetypes are based on the best scientific understanding of disease. When you understand the pathophysiology of Still’s, or Cat Scratch, or any disease and are confronted with a patient who may have a given disease, it is good to have an archetype to which you can refer and apply the pathophysiologic understanding. It puts a face on the disease.
3) Anecdotes used as an example of a general case
This has a long tradition in medicine. I need to give a lecture about Zoonosis (animal related infections). Often the lecture will start with a case presentation. It may be a real case, or it may be a combination of several cases. The purpose of the anecdote is to tie the lecture, which is often filled with the dry results of clinical studies, to a real person with a real disease. Unlike the archetype, which is used to help understand unusual diseases, the anecdote is used to describe common medical conditions and is codified in the case presentation. Anecdotes are effective tools to aid learning about a common disease.
4) Anecdotes used as an example of atypical manifestations of common disease.
We all get blindsided by common diseases that present oddly. Classic angina presents with pain that radiates into the neck and down the left arm, often brought on by exercise. As a resident I had a patient whose angina was left wrist pain with exercise. I like to tell that anecdote to residents to emphasize the need to be on the lookout for atypical manifestations of common diseases. Medicine would be so much simpler if patients would read the textbooks before coming to the hospital.
Corollaries include anecdotes used as examples of typical manifestations of uncommon disease (a cousin of the archetype anecdote) and the anecdote used as examples uncommon manifestations of uncommon diseases. All of these disease manifestations are best remembered through anecdotes, and serve as cautionary tales. If you are not careful, you too may be fooled and miss an important disease.
5) Anecdotes used as an example of proven concepts
Syphilis has a wide variety of manifestations and is best treated with penicillin. Proven concepts. When I teach a resident or discuss a case, I may use an anecdote about one of the many syphilitics I have treated over the years.
Homeopathy cannot work for any disease except thirst, since all homeopathic concoctions contain nothing but water. A proven concept in so far as it is possible to prove a negative. I could present a case where a patient used a homeopathic remedy for syphilis and the disease progressed unhindered by the homeopathic concoction.
Both the efficacy of penicillin for the treatment of syphilis and the lack of efficacy of homeopathy are proven concepts, illustrated by anecdotes.
6) Anecdotes used as an example to prove an unproven concept
This would be an example of the justly derided anecdotal evidence, and differs from the other, perhaps more benign, forms of anecdotes.
Homeopathy does work, here is the proof: I gave a patient with joint pain a homeopathic concoction and the patient improved. The knee pain went away. Proof of the efficacy is in the successful intervention.
It is here that the motto “the plural of anecdote is anecdotes not data” is most applicable. It is whether the anecdote is being used to prove a concept, or used as an example of a proven concept is where practitioners of CAM and real medicine differ, both on how the anecdote should be used and the validity of the anecdotes use.
It is the use of anecdotes as a proof of concept upon which the basis of much of CAM rests, and is the most unreliable use of anecdotes. Given the vagaries of human memory, the difficulty in proving causality, and the lack of biologic and physical possibility of most CAM interventions, using anecdotes as proof of CAM effectiveness is a particularly problematic use of anecdotes. Even if Dr. Weil tarts them up by calling them uncontrolled clinical observations.
Saying “the plural of an uncontrolled clinical observation is uncontrolled clinical observations, not data” just doesn’t seem as catchy.
It is this kind of anecdote that has the most power, both in CAM and in the reality-based practice of medicine. An anecdote where something happened has far more power to convince than an anecdote where nothing happens. A patient has an adverse reaction after a vaccine or gets better after taking echinacea. The anecdote suggests causality. If you respond that clinical studies demonstrate the lack of data to support their anecdote, they will look at you like a dorky brainiac divorced from the real world. If you counter with an anecdote where, under the same circumstances, for you nothing happened after the vaccine or the echinacea, you will probably get the same “man what a clueless dork” roll of the eyes. Hits have far more power than misses, your anecdote will be dismissed, and you will be branded a hypocrite since in the past you have called anecdotes worthless.
Skeptics of CAM are doomed to forever lose against the formidable power of the positive anecdote, as one positive anecdote is worth 1000 studies and 10,000 negative anecdotes.
Can anecdotes ever be considered proof? There may be times when anecdotes can be definitive without the need for classical clinical trials (3). Some types of adverse drug reactions may be considered proven with anecdotes. In “Anecdotes that provide definitive evidence” the authors provide 4 instances that they feel anecdotes are sufficient data to prove a hitherto unproven idea.
1) Extracellular or intracellular tissue deposition of the drug or a metabolite
2) Specific anatomical location or pattern of injury
3) Physiological dysfunction or direct tissue damage that can be proved by physicochemical testing
4) Infection as a result of administration of a potentially infective agent or because of demonstrable contamination.
These are all very narrow instances of anecdotes as proof, and may be unreliable, as case reports for adverse drug reactions are rarely confirmed in subsequent studies (4).
I would consider a stroke immediately following cervical spine manipulation in the same category where the anecdote is convincing. I am sure others will disagree or have other anecdotes they consider definitive. What level of plausibility where an anecdote could be considered sufficient proof to confirm a hypothesis is uncertain, depending on prior plausibility and will vary from person to person.
But as proof of a concept in CAM, in most instances one has to combine aphorisms: “the plural of anecdote is anecdotes, not data” and “extraordinary claims require extraordinary evidence.”
6) Anecdotes used to guide treatment when there is no data.
Grand Rounds this week was on Pulmonary Hypertension, a rare disease with an array of new, effective, and very expensive therapies. Pulmonary hypertension has a variety of causes and stages. When do you start therapies and in what order. There is some data, but given the rarity of the disease, there are many instances where the particular case does not match the known information. So as a clinician you rely on the anecdotes of others and your past experience to guide therapy.
The same issue occurs in AIDS treatment. There are 28 different HIV medications and more coming. There is some data to suggest the best initial therapy, but as resistance and intolerance occurs to the medications, often it is anecdotal data that determines which medications are used.
Clinicians get comfortable with one therapy and perhaps had poor outcomes with other therapies, and these anecdotes influence further interventions. Unlike CAM, there is biologic plausibility behind the anecdotes for your interventions. You always understand that using anecdotes to help decision making is a temporizing measure because, and again unlike CAM, there will be future studies that will help guide optimal therapy.
7) Anecdotes used as the history of the present illness
People love stories. They love to tell stories, especially about themselves and about their diseases. Patients do not give organized medical histories. They tell extended anecdotes about their illness. What a patient tells me about their disease and what really happened are often two different issues and all too often (especially when it is the last consult on a Friday) the patient is particularly loquacious with the details of their disease that have no bearing of the diagnosis. Part of being a doctor is learning to extract the pertinent information from the sometimes endless detail in the anecdotes.
8) Anecdotes used to provide understanding of the unknown.
Often the diagnosis and treatment of a disease can be uncertain, and by referring to themselves and others with anecdotes about the process, it puts the illness in a context that renders it explainable to the patient and their close friends and family. Patients frequently ask my experience with their disease, and I used anecdotes about past successes and potential pitfalls to explain and personalize their disease. The anecdotal experiences are, I think, more reassuring and understandable, than just recitation of the statistics of their disease, although I provide that as well. Patients don’t care that 98% of people with streptococcal endocarditis are cure, since for them it is all or nothing. So I mention the success rate, the outcomes, and allude to the experiences I have had with other patients.
9) Anecdotes used to guide diagnosis or therapy because of either prior unusual outcomes, good or bad.
This is the hard one, as a clinician, to be conscious of. If you have ever had a particularly horrible outcome from the care of a patient, you will remember that case. Vividly and forever. When you have a similar case, that anecdote will float though your head and may alter your approach, perhaps inappropriately so. Bad outcomes and complications have enormously greater impact on care than good outcomes, and I bet most of us made rounds with the ghosts of patients past, who look over our shoulders and guide our therapy. These anecdotal ghosts can exert more power than evidence, especially when the data is sparse or the outcome was particularly bad.
It is rare for anecdotes to be used in a single context. Language is not precise and anecdotes can have many interpretations. Anecdotes are flexible and one anecdote can have many functions to the user and many interpretations by the listener. Perhaps we should be more precise in our use of anecdotes. When professional societies make recommendations, the strength of the data has a rating system, of which there are a variety (5).
Perhaps, and I am only half serious when I suggest this, we need a similar classification for anecdotes when we discuss medicine and, especially, CAM.
Someone tells an anecdote about how CAM works or how it fails. How is it being used? After the anecdote, put the type of anecdote in a parenthesis, just like they big boys do when making recommendations. Then we can assess the validity of the anecdote in the context the writer intended, rather than the context applied by the reader.
Type 1) Anecdotes used to describe new observations.
Type 2) Anecdotes used as archetype.
Type 3) Anecdotes used as an example of a general case.
Type 4a) Anecdotes used as an example of atypical manifestations of common disease.
Type 4b) Anecdotes used as examples of typical manifestations of uncommon disease.
Type 4c) Anecdote used as examples uncommon manifestations of uncommon diseases.
Type 5) Anecdotes used as an example of a proven concepts.
Type 6) Anecdotes used as an example to prove an unproven concept
Type 7) Anecdotes used as the history of the present illness.
Type 8) Anecdotes used to provide understanding of the unknown.
Type 9a) Anecdotes used to guide diagnosis or therapy because of either prior unusual outcomes, good.
Type 9b) Anecdotes used to guide diagnosis or therapy because of either prior unusual outcomes, bad.
I have an anecdote that suggests such an approach will work (Anecdote Type 1).
1) Budget: archaic a quantity of material, typically that which is written or printed. An old classic is a Budget of Paradoxes.
2) Like complementary, which complements nothing, and alternative, which isn’t, a ‘proving’ proves nothing.
3) Anecdotes that provide definitive evidence 2006;333;1267-1269 BMJ . Jeffrey K Aronson and Manfred Hauben
4) BMJ. 2006 Feb 11;332(7537):335-9. Case reports of suspected adverse drug reactions–systematic literature survey of follow-up.
5) BMC Med Res Methodol. 2006; 6: 52. A system for rating the stability and strength of medical evidence
9 Responses to “A Budget of Anecdotes”