Articles

Anecdotes: Cheaper by the Dozen

A loan officer sets up a meeting with an aspiring entrepreneur to inform him that his application has been denied. “Mr Smith, we have reviewed your application and found a fatal flaw in your business plan. You say that you will be selling your donuts for 60 cents apiece. “Yes” says Mr. Smith, “that is significantly less than any other baker in town. This will give my business a significant competitive advantage!” The loan officer replies, “According to your budget, at peak efficiency the cost of supplies to make each donut is 75 cents, you will lose 15 cents on every donut you sell. A look of relief comes over Mr. Smith’s face as he realizes the loan officer’s misunderstanding. He leans in closer, and whispers to the loan officer “But don’t you see, I’ll make it up in volume.”

If you find this narrative at all amusing, it is likely because Mr. Smith is oblivious to what seems like an obvious flaw in his logic.

A similar error in logic is made by those who rely on anecdote and other intrinsically biased information to understand the natural world. If one anecdote is biased, a collection of 12 or 1000 anecdotes multiplies the bias, and will likely reinforces an errant conclusion. When it comes to bias, you can’t make it up in volume. Volume makes it worse!

Unfortunately human beings are intrinsically vulnerable to bias. In most day to day decisions, like choosing which brand of toothpaste to buy, or which route to drive to work, these biases are of little importance. In making critical decisions, like assessing the effectiveness of a new treatment for cancer, these biases may make the difference between life and death. The scientific method is defined by a system of practices that aim to minimize bias from the assessment of a problem.

Bias, in general, is tendency that prevents unpredjudiced consideration of a question (paraphrased from dictionary.com). Researchers describe sources of bias as systematic errors. A few words about random and systematic errors will make this description clearer.

Random Error

Random errors are unpredictable variations in a measurement. Random errors, by definition are unpredictable in how they may affect any single measurement. For small samples, random errors may lead to an incorrect conclusion. For instance, 4 consecutive “heads” or 4 consecutive “tails” coin flips are not rare, but when they occur, may give the false impression of an unfair coin. Larger samples decrease the likelihood that random errors will result in an errant conclusion. 1000 coin flips will rarely deviate very much from a 50/50 distribution. More data increases the confidence that a sample measurements approximate the true value.

Systematic Errors (a.k.a. bias)

Bias is a non-random (or systematic) error which tends to distort results in a particular direction.

Larger sample sizes are not protective against bias. In fact, larger sample size increases the likelihood that bias will lead to an erroneous conclusion. Lets say that we want to measure the birth weight of a population of inner city newborn babies. Our scale is mis-calibrated in a way that 8 ounces is added to the weight of every child. The will result in a systematic over estimation of the birth weight of the babies. If we weigh lots and lots of babies, we will develop very impressive statistics with very tight standard errors around the mean weight, but they will be wrong because our measurement is biased. If we do not recognize the bias, more data will make us more confident in our data, but will not make our data any less wrong!

Bias is the nemesis of researchers, and is difficult, if not impossible to eliminate completely. The best researchers strive to minimize bias in their study designs, and acknowledge potential sources of bias they cannot eliminate.

Some types of data are more prone to bias than others. The vulnerability to bias is one of the most important qualities in determining the reliability of data derived from different sources of evidence. At the bottom of the hierarchy of reliability are the anecdote and its first cousin, the testimonial.

Anecdotes are narratives of one time events. Because they are singular events, anecdotes of anomalous occurrences are not balanced by averaging effects that results from multiple observations, thus they are highly vulnerable to random errors. So, if we look at enough anecdotes, the errors will neutralize each other, and we can create a true impression of nature, right?……..Wrong! If a collection of anecdotes represented a random sample of independent events, such a strategy might work, but collections of anecdotes are rarely random or independent….quite the opposite is true. Because of the biases intrinsic to anecdotes, a collection is even more likely to reinforce a false conclusion.

The only anecdotes that become part of the collective consciousness are those that are remembered by an observer, deemed worthy of repetition, and then actually communicated by the spoken or written word. There are many factors which influence the recall, interpretation, and reporting of experience. These factors are not random.

One of the principle reasons experiences are recalled and repeated is precisely because they are unexpected or out of the ordinary. Anecdotes of mundane events are unlikely to be repeated. Great significance is often attached rare and unexpected event, and the human mind seems programmed to look for “causes” of these events. Much of the time the events are just rare, but random occurrences (point examples of random error).

Confirmation bias is a tendency to notice and attach significance to information that reinforces ones preconceptions, while dismissing or ignoring contrary information. Confirmation bias has a great influence on which events are remembered and repeated as anecdotes.

Anecdotes are greatly influenced by the image the narrator has of him or herself, and even more so the image he/she wishes to project to the outside world. I call this Bumper Sticker Bias. I regularly see bumper stickers boasting that the driver’s progeny is an honor student a their respective school, but, strangely, I have never seen a sticker proclaiming “My kid was expelled from Central High School”. Bumper stickers are not random expressions of human experiences, and neither are anecdotes.

“Case Reports” are anecdotes enshrined in the medical literature. There is reason to believe that case reports are somewhat more reliable than an anecdotes told by your taxi driver or your cousin (at least when it comes to medical information). Physicians are ostensibly trained observers, but are in no way immune to all the biases ubiquitous to the human condition. For a case report, there is a written document (the medical record) to consult to for re-creation of the narrative. Most journals publish very few if any case-reports and credible journals do so only after peer review. Even published case reports are not given much weight by the medical community, and the articles are infrequently cited in future literature. Case reports can, however, have value. Rabies, for example, is so uniformly fatal, that a single example of an unvaccinated patient surviving infection with rabies is a highly significant case.

There is a familiar skeptical mantra that says “the plural of anecdote is not data.” The original attribution, and even the accuracy of this quote is a little uncertain. I would argue that the plural of anecdote is data; they are just really unscientific, really really unreliable data. Under the right conditions, anecdotes may generate hypotheses. For these hypotheses to be widely accepted, they must be confirmed by more robust data gathering and analytic methods. One of the recurring themes of pseudoscientific medicine, is that hypotheses generated by anecdotes are confirmed by nothing other than an avalanche of more anecdotes. Like the failed entrepreneur introduced at the beginning of this post, the purveyors of these treatments think that augmenting anecdotes with more anecdotes validates their hypotheses. But they are wrong. When it comes to bias, you can’t make it up with volume.

Posted in: Clinical Trials, Science and Medicine

Leave a Comment (31) ↓

31 thoughts on “Anecdotes: Cheaper by the Dozen

  1. bhami says:

    Please help us understand where to draw the line on requiring rigorous proofs such as randomized clinical trials. E.g. isn’t it more than an “anecdote” when I infer that the sun will rise tomorrow, based on my seeing it rise previously? And doesn’t each physician’s body of personal experience with thousands of patients count for something?

  2. UncleHoot says:

    This is a very thought-provoking article.

    I wonder if this would also tie into the concept of a medical consensus. Simply getting (nearly) everyone to agree on a treatment/diagnosis does not necessarily make it the the best choice, especially if those making the decision share the same underlying bias. Understandably, sometimes that may be the best we have, but perhaps it’s important to differentiate.

  3. nybgrus says:

    @bhami:

    It is just a question of how certain we are of an outcome.

    For your example of the sun, having seen it rise in exactly the same way and time for years would provide a certain amount of comfort that it will happen again the next day. But if that is the only evidence we had, we would be forced to say that to the best of our knowledge the sun will rise, but we can’t be so certain because it is only our anecdotal observation.

    But we have many lines of converging evidence to support our anecdotal experience – understanding of the solar system, laws of physics, etc etc all give us a very high certainty.

    Of course, all of this should be taken in context of both how much precision we need and how much we care. For example, what if for your entire life just by chance you never saw a solar eclipse. Someone else has seen a number of them. Someone else has lived his or her entire life north of the Arctic circle and only knows 6 months of night and 6 months of day. Now you each have anecdotes – how will you settle this? If it matters not to your life, or if it is just an issue between the solar eclipse or not, you may write it off and simply say that the error of 2-5 days per year just doesn’t matter.

    But if you really needed to settle it, you can’t use anecdote to settle the issue since each of these 3 people have vastly different experiences… each of which is fundamentally biased. And that is the point of this post. We need something more than anecdote – even a lot of them – to settle the issue.

    As to how much evidence we “need” well, that all depends. How much precision do we need? That depends and is often a judgement call. Also, depending on the question, different types of evidence (RCT, sytematic review, bench science, retrospective review, ect) may be better than others and some may simply not be possible (RCTs of smoking effects on cancer would be unethical, for example). But from a scientific standpoint, we want as much evidence and precision as can be reasonably attained, so we always go for more.

    As for the thousands of patients of experience… yes, it does count for something just not as much as most people (including physicians) would think. The real utility of the experience is to be able to recognize presentations of disease. The reason that I, for example, would be less able to practice medicine than an attending (besides the obvious differences in knowledge) is that I may know that a “lacy reticular rash” is part of the presentation of parvovirus B19 infection, I simply woudn’t have the confidence to accurately identify which rash is “lacy reticular” and what features may or may not be present (where to look and what for) to confirm or disconfirm this finding. The other aspect of the physician’s experience that is useful is in hypothesis generation. Dr. Jay Gordon, for example, has decades of experience with literally thousands of patients which have led him to believe that vaccines cause autism. That is a hypothesis based on the experience he had – a claim. But that is not evidence that it is acually happening. Which is why we studied it and determined his experience was spurious and incorrect. Which is once again the point of this post.

    So in sum, rigorous proof is not equivalent to RCT. Convergence of multiple lines and types of evidence is rigorous evidence. Even a single, well designed, well powered, RCT is not “rigorous proof” though for obvious reasons it will be much more highly valued. And in the absence of or inability to provide more evidence, we may act on it; but this would be with the knowledge that it is provisional. The reason why something like evolution is so incredibly unassailable is because of how voluminous and varied the evidence is, all pointing to the same conclusion – not because we have a single bit of “rigorous proof.”

    Hopefully that helps make it more understandable

  4. Ed Whitney says:

    Allow me to jump in early to ask for people’s insights about a rather important current topic, namely whether epidural steroid injections (ESI) should be done for back pain with or without nerve root signs. The multistate outbreak of fungal meningitis from contaminated lots of methylprednisolone has sort of put these injections in the public eye, but at the Annals of Internal Medicine there is a meta-analyses of randomized clinical trials of ESI for sciatica which concludes that they provide only small, clinically unimportant short term relief of leg pain, no relief of back pain, and short term, clinically unimportant improvements in function and disability. Long term there is no effect on any of these endpoints.

    Since ESI is a big ticket item for Medicare and for private insurance, its advocates are likely to push back against this meta-analysis with numerous anecdotes from their practice, when they performed the injections and were met with success. There is likely to be pressure from payers to limit the use of ESI in the people they cover in their policies.

    The defenders of ESI are likely to insist that there are subgroups of patients, not tested separately in the randomized trials, who benefit from an intervention which does not have a statistically significant benefit in radicular back pain patients in general.

    I suspect that this meta-analysis will prove controversial when it comes to general attention. Interesting questions about the nature of randomized trials will be raised. The one I am interested in right now deals with this area of specific patient subgroups being likely to benefit from treatments which fail on the measures of central tendency which meta-analyses focus on: means, standard deviations, odds ratios, hazard ratios, and the like. The ball will be in their court to show which these subgroups are, but I feel quite certain that these arguments will be made by the proponents of ESI.

    Here is the question I hope someone will comment on: statistical tests are mathematical constructs which study the distribution of random variables. Does this mean that they assume that the responses of patients in a clinical trial are random variables? And is this equivalent to assuming that all variation between participants in a clinical trial are due to random chance alone? I think so, but am not certain. The propositions seem logically equivalent but perhaps there are nuances that qualify this logical equivalence.

    Any insights are appreciated!

  5. BillyJoe says:

    Harriet:

    This is off topic, but the comments on old articles seem to be disabled so I will post a brief comment here.

    You wrote an article about Screen For Life in 2008.
    The company offering the four screening tests has now started operating in my home state of Melbourne, Australia. I know because I received a personalised invitation in the mail. The first thing I did was google it and your article came up under Screen For Life Scam. I should have come here first of course.

    Thanks for that article.
    I’m not one for doing screening test anyway (I know there is good evidence to do some tests, though often they are of marginal benefit vs risk, but it does not fit in with my personal outlook on life), but I have all the reasons for not having those tests to pass on to family and friends who, no doubt, will receive these invitations also.

  6. WilliamLawrenceUtridge says:

    Historically, RCTs are more prominent now because the low-hanging fruit have been plucked. You don’t need an RCT when you’ve got a hard end-point with objective measures and a clear ability to distinguish between groups. If something is universally fatal, but everyone who receives a treatment universally survives, you don’t need a randomized controlled trial. The rabies example in the article is useful here. Rabies kills pretty much everyone infected. Any treatment that lowers the death rate below 100% is easily recognized as effective.

    There’s not many diseases and treatments that meet these criteria anymore though – because they were the function of the earliest efforts of medicine. CAM mostly deals with self-limiting symptomatic complaints with few objective measures, so more than most conditions or treatments, you have to have good controls.

  7. daedalus2u says:

    There are several types of stories that are called “anecdotes”.

    The first type is a story that is factually accurate in every detail, but the data was all recorded after the event or retrospectively or with very small n, or even n=1. This is an anecdote, but it is also data of limited statistical significance.

    The second type of “anecdote” is a made-up story that is not a description of a unique event, but is a factually accurate hodge-podge of multiple factually accurate stories blended together. This is an anecdote, but it is not data. It may be used for hypothesis generation.

    The third type of anecdote is the sharp-shooters’ fallacy. The story may be factually accurate, but it is the one remarkable and atypical story out of hundreds or thousands of different results and is very likely atypical due to unknown information. This is the type of testimonial that many CAM practitioners rely on. This might be a positive biopsy that was then “cured” by some type of CAM treatment, where in fact the biopsy by itself was curative. It might be a false positive diagnosis that was then “cured” by homeopathy.

    The fourth type of “anecdote” is a made-up story that is not factually accurate. This is not data, it is not useful for hypothesis generation, it is only useful for selling snake oil, i.e. fraud.

    In no case do positive anecdotes reduce the prior plausibility of an intervention. In the case of anecdotes that are completely made-up and totally false, they have no effect on the Bayesian prior plausibility. Just the way that the assertion that 2+2=5 has no effect on the Bayesian prior plausibility of anything.

    To answer Nybgrus’ question, how much data is needed, it depends what the data is needed for. If the need is for a differential diagnosis, how much data is needed to prescribe a differential treatment. If the treatments for the two conditions are “the same”, there isn’t a great need for a differential diagnosis.

    To get very high confidence numbers, you need to have a theoretical basis. For the Sun rise example, if by age 40, you have seen the Sun rise every day, you have directly witnessed ~15,000 Sun rises. That puts a limit of about less than one non-Sun rise in 15,000 days. If you think the Earth is 6,000 years old, then the direct limit is ~ 1 in 2 million days. If you know the geological record, the direct limit is ~1 trillion days. If you know the heliocentric model of the solar system, Newtonian mechanics, conservation of momentum, you know that the Earth can’t stop rotating without a very large momentum transfer that could only come from contact with a mass large enough to essentially destroy it.

    I don’t need any Sun rise observations to understand the heliocentric solar system model and be able to assert that the Sun will rise tomorrow with essentially absolute certainty because anything that would prevent that would also destroy the Earth.

    EBM is like getting up every morning before dawn for many years, recording the Sun rise and projecting that it will rise tomorrow. SBM is like also adding the heliocentric solar system model and appreciating that the few observations of the Sun not rising were due to volcanic ash making the atmosphere opaque.

  8. ConspicuousCarl says:

    [...]case study[...]

    Another use of case studies which should be distinguished from anecdotes is as examples in education. A case study included in a book is presented not as evidence, but as a typical example of a patient which is known to be typical based on larger existing data. It’s an education tool, much like a metaphor can be a handy way to teach established concepts but is usually a disaster as a basis for an argument.

    David Weinberg said:
    I would argue that the plural of anecdote is data; they are just really unscientific, really really unreliable data.

    Or maybe it is actually really good data for a different hypothesis. My personal definition of a confounding factor is a thing you are measuring when you think you are measuring something else. Your results might be crappy data for the question “does homeopathy work?”, but it might be really good data to answer the question “how much are respondents willing to lie in order to make Andrew Weil feel like he isn’t wasting his life?” Similarly, a collection of memetic anecdotes actually might be a pretty good way to assess which qualities a story must have in order for it to be passed along. No such thing as bad data, only bad data-hypothesis combinations.

  9. ConspicuousCarl says:

    bhami on 21 Nov 2012 at 1:02 pm
    isn’t it more than an “anecdote” when I infer that the sun will rise tomorrow, based on my seeing it rise previously?

    Collecting data consecutively can actually serve as a pseudo-randomization so long as the exact time (or day) is not likely to be an influencing factor. Those observations also aren’t anecdotes because, except for extreme situations, your knowledge of the sun rising yesterday (and every other day) did not rely on whether or not someone chose to pass the information along to you. You are limited to a particular period (your life), but on the other hand you aren’t missing any points within that period. So you can’t say that the sun will never fail to “rise”, but you can say that it seems likely to rise consistently in the absence of some other influence because it has risen every single day it has had the chance.

    This is why Andrew Wakefield was such a bastard for claiming that the patients included in his paper were “referred consecutively”, when in fact they were selected and he probably passed over a lot of other kids. It’s not as good as randomizing, but in the absence of some deliberate or shared motivation for choosing that exact time, the fact that several people with the same combination of illnesses arrive in a row carries some weight. If they were uncommon, it would be unlikely that they would all just happen to decide to schedule doctor visits that week.

  10. ConspicuousCarl says:

    Um, I guess it should be noted that the reason why we are currently so much more certain that the sun will rise tomorrow (vs. a mere lifetime or observation) is because we now know why is does what it does at all, and how much it would take to change that.

  11. nobeardpete says:

    Regarding the sun rising analogy, it’s important to remember that one of the important differences between a pile of anecdotes and a reasonable set of data is that the anecdotes are not collected in a systematic way. If you had thousands of people write you an email with a story that goes, “Yeah, I saw the sun come up one day,” it would be a collection of anecdotes of little worth, because it’d be consistent with a situation in which the sun occasionally doesn’t come up, but in which people are more likely to discuss instances of the sun coming up.

    In order to avoid falling into this sort of trap, we often choose to consider a set of observations according to some straightforward criteria that would not be expected to predispose our observations towards only certain outcomes. In forward-looking sciences, the best way to do this is to come up with a set of criteria ahead of time, and to only then perform the experiments or make the observations. However, when this is not possible, another widely accepted and generally quite good means of selecting observations is to take a comprehensive record of every event in question, and to consider a certain number of consecutive cases.

    If we’re wondering whether or not the sun will come up tomorrow, I think we can consider a case series of the past, say, 10,000 consecutive days (some 27 years) to be a reasonable set of data. The fact that we have a simple, rigorous selection criterion (every day for the last 10,000 days, no skipping, not just the days that have been brought to our attention by third parties, etc, etc) is what makes this data as opposed to a collection of anecdotes. And, looking at this data set, we can easily see that the sun seems to come up 100% of the time. Which lets us say with some confidence that the sun is likely to come up tomorrow.

  12. nybgrus says:

    @ carl and nobeardpete:

    Excellent additions and refining to what I had said.

    The singularly best point is that watching the sun come up every day for [X] amount of time is not quite the same as anecdote.

  13. nybgrus says:

    @uncle hoot:

    absolutely you are right. The premise, however, is that a number of people, all well versed in the relevant sciences would mitigate the various biases inherent to interpretation when coming to a consensus. This is why guidelines and consensus statements tend to be much more conservative than one might otherwise think. Of course, anybody who blindly puts faith into any person, group, organization, etc is not being as rigorous as he/she should. For example, I trust SBM quite a lot… but only because I spent a lot of time and effort vetting the authors and what they write. But I always assume they can be fallible and when challenged or if the article is contrary to what I think I know, I take the extra effort to vet it on my own.

  14. Janet says:

    Who is David Weinberg? Nothing in the “About” page.

  15. weing says:

    from about the author on his first post about acupuncture for amblyopia

    “David Weinberg is a full-time academic vitreoretinal surgeon, and professor of ophthalmology at the Medical College of Wisconsin, Milwaukee. He completed a fellowship in vitreoretinal diseases and surgery at the Harvard Medical School, and has authored or co-authored over 70 publications in the peer-reviewed literature and 10 book chapters.”

  16. Mika says:

    I always though the quote was “the plural of anecdote is not evidence”.

  17. pmoran says:

    CC: Or maybe it is actually really good data for a different hypothesis

    – or, anecdotal evidence may be adequate for some practical medical purposes, e.g. where there is no known effective treatment for a serious condition but anecdotal evidence that X has helped in one or more cases. While not common within the mainstream, this is how much of CAM works. It should be seen as having a modicum of internal validity for these unsophisticated, or sometimes merely very desperate, people.

    I would go so far as to suggest that behind all medical knowledge there is an unspoken “for present practical purposes” which is partly why we in truth operate under a fairly wide range of types and quality of evidence.

    We aspire towards, and commonly feign, that we are using only the most reliable and uniform standards of evidence but are hampered in that by costs and practicalities — and a little self-delusion, if it is as true as some claim that “most published research findings are wrong” and Ben Goldacre’s grim appraisal of the pharmaceutical industry is correct (see Scot Gravura’s article) is correct?

    Let’s make sure we fully understand ourselves before fine-tuning our levels of holier than thou-ness with CAM. We have ample, if anecdotal, experience of how that just doesn’t work.

  18. Jan Willem Nienhuys says:

    Regarding the sun rising analogy, it’s important…

    If you have observed something quite frequently (I doubt if many people often see the sun rise, especially in cloudy countries or countries where the sun often rises at 4 or 5 am or both) it doen’t mean it is real. In case of the sun we have as additional evidence that after dark it gets light every day, even if we don’t actually see the disc of the sun rising above the horizon. Also we notice that the sun moves with a very uniform speed across the sky and we notice that its period of absence suggests that it is moving with the same speed when we can’t see it. This suggests that it always moves with that speed, and doesn’t appear and disappear suddenly or make jumps. But many other people also saw the sun rise. That is really important.

    Suppose you meet a person who claims: “almost every night I leave my body to roam about the world and visit even the spiritual beyond”. Should we think this is reliable information because this person experiences it even more often than I actually see a sunrise (i.e. watch the sun’s upper edge pass the horizon and move upwards until the full disc is in view)? Of course not. So apparently there is much more needed than the certainty ‘I experienced this or that quite often’.

  19. ConspicuousCarl says:

    pmoran on 22 Nov 2012 at 3:45 pm

    CC: Or maybe it is actually really good data for a different hypothesis

    – or, anecdotal evidence may be adequate for some practical medical purposes, e.g. where there is no known effective treatment for a serious condition but anecdotal evidence that X has helped in one or more cases. While not common within the mainstream, this is how much of CAM works. It should be seen as having a modicum of internal validity for these unsophisticated, or sometimes merely very desperate, people.

    Yes, but, as already explained, that is only true in the cases of diseases which have 0% natural/random remission or improvement.

    And that is not how any CAM “works”. The way CAM works is by offering fake treatments for which the above usefulness of anecdotes does NOT apply, which is to say highly subjective, variable, and self-limiting conditions like back pain and flu.

    and a little self-delusion, if it is as true as some claim that “most published research findings are wrong”

    Hmmm… a bit surprising to hear you go quite that far. If you read this blog regularly, you darn well ought to know the misleading reality behind that quote. It’s not self-delusion of scientists or skeptics, it’s the self-delusion of those “unsophisticated” people you were talking about earlier. Everyone here (except for you maybe?) knows that there is a continuum of quality level in the world of research, and we don’t jump to conclusions every time some small-to-medium study claims to show something interesting.

    and Ben Goldacre’s grim appraisal of the pharmaceutical industry is correct (see Scot Gravura’s article) is correct?

    And then this. You are just dipping into the old straw man here. Not once, EVER, has anyone on this website held the opinion that “the pharmaceutical industry” was to be trusted to undue lengths. Nor does that excuse the same (if less discrete) BS tricks pulled by CAMers. You make excuses for CAM, but then try to lynch us as delusional worshipers of Big Pharma, which we are not.

    our levels of holier than thou-ness

    Drama alert.

  20. Mark P says:

    All those people who supposed – after a lifetime of experience – that the sun would rise again were, sadly, incorrect.

    Their incorrect understanding has embedded itself in our language pretty comprehensively, but it’s still the wrong way of thinking about what is happening.

    The sun doesn’t rise. The earth rotates.

  21. nybgrus says:

    Very true. Plus, in about 4.5 billion years all that anecdotal experience will not prepare you for what will happen, regardless of whether or not correct terminology is used.

  22. Great article … and hit in me in the face as I was perusing Consumer Reports’ health section and came across this wonderful example of how anecdotes can be used improperly to promote quackery:

    http://www.consumerreports.org/cro/2012/04/alternative-treatments/index.htm

    Don’t know if you need a login to get there, but the headline pretty much says it all –

    Alternative treatments: More than 45,000 readers tell us what helped

    And if you read the article you get a lot of essentially useless statistics of people who have self-reported the wonderful impact of the alternative treatment modality that they chose when traditional med failed them. They start the woo off with a chart showing that over 40% of people that received chiropractic for allergy treatment were “helped a lot”. It goes downhill from there …

    Would love to see this consumer reports article in it’s entirety torn apart here.

  23. David Weinberg says:

    Sorry for the late response. Due to the holiday, I have been offline for awhile.

    Ed Whitney asks:

    “statistical tests are mathematical constructs which study the distribution of random variables. Does this mean that they assume that the responses of patients in a clinical trial are random variables? And is this equivalent to assuming that all variation between participants in a clinical trial are due to random chance alone?”

    This really gets to the heart of the problem. Patient responses in any setting are complex and influenced by a many factors. Depending on the disease, and the specific endpoint being studied, some, but surely not all, of the factors that influence a patient response may be known. Since many factors cannot be identified, and interact in unpredicable ways, they, for all practical purposes are random. In a clinical trial, we are trying to sort out the experimental treatment effect from all the random noise. By randomly assigning patients to treatment groups, the confounding variables become distributed equally between the treatment groups, so that the effects of the experimental treatments can be isolated.

    If the randomization is effective, and the trial is free of bias, then a difference between the groups is interpretted to be due to the experimental treatment. In such a situation the differences between groups are not random, but due to a systematic influence of the experimental treatment.

  24. Andres says:

    There has been some research on the effectiveness of enough intramuscular ascorbic acid (100mg/kg twice a day for 7 days) on rabies (guinea pigs): Prevention of rabies by vitamin C, S. Banic (full text).

  25. Oh, another proponent of vitamins as panacea. The research mentioned by Andres is over 30 years old.

  26. Andres says:

    Well, there seems to be even older papers (in vitro?) about it: reference 12 of Junglebut’s “Studies on the Inactivation of Diphteria Toxin by Vitamin C (l-ascorbic acid)”.

    A search on Google Scholar doesn’t give rise to any recent research except dealing with rabies virus inactivation in vitro (8 years ago).

    I haven’t found any refutation either.

    Looking away of promising preliminar evidence and ignoring it isn’t Science.

  27. nybgrus says:

    “promising preliminary evidence” without follow up is not worth very much. A lack of follow up usually indicates the preliminary evidence wasn’t that promising. Coupled with the massive amount of literature demonstrated no other “unusual” uses for VitC, plus some knowledge of the biochemistry and physiology of it, allows for an informed reader to discount these very old preliminary studies. That is, in fact, scientific.

  28. Chris says:

    Andres, diphtheria and rabies are nasty diseases, with very high death rates (10% and over 90% respectively). You definitely need something more definitive that little old studies, especially just in a petri dish (in vitro), to skip the vaccine for those supplements.

  29. Andres says:

    @nybgrus: The first reference was an experiment with guinea pigs (no ascorbic acid production in their livers just like us): death rate of 17/48 with intramuscular 100mg/kg twice a day of vitamin C for 7 days versus 35/50 without treatment. The difference in outcomes seems significant (a little aside: I am now trying to grasp conditional error probabilities given that I am favoring Fisher’s interpretation of this kind of experiments, but a quick and dirty computation of the probability of those two numbers or more apart ones coming from the same death probability in both arms of the experiment was quite low). It is not clearly irrelevant to me. I don’t think that we may be certain that willful ignorance hasn’t been at work on the dismissal of the potential of high enough doses of ascorbic acid. So to me if it has not been experimentally refuted then it still stands.

    @Chris: Of course I am not advocating skipping vaccines for serious diseases. Even if a treatment were proved, it wouldn’t be clearcut their worthlessness.

Comments are closed.