Articles

When Further Research Is NOT Warranted: The “Wisdom of Crowds” Fallacy

Most scientific research studies have at least one thing in common: the conclusion section ends with, “further research is warranted.” I’d say it’s about as common as the “talk to your doctor” disclaimer in TV ads for pharmaceutical products. And in a way, they both serve the same purpose. They’re a “CYA” move.

What does “further research is warranted” mean in plain English? I think it can be roughly translated: “My research study is not of the size or scope to fully explain all the phenomena described in this article. Therefore, draw conclusions beyond the data and study methods at your own risk. And yeah, my work is important and cool – so people should study it further.”

Of course, the first two sentences are reasonable – we should always remember not to draw conclusions beyond the information provided by the data we’ve collected (even though that’s about as challenging as getting a beagle not to eat a table scrap in an empty room). The real problem is the third sentence. Is the research promising enough to require further investment? How are we to know if further research is indeed warranted? I would argue that it should not be based solely on the subjective opinions of the researchers nor the popularity of the research topic to the general public.

As my colleagues here at Science Based Medicine have already explained, plausibility is an often overlooked but critical piece of the value proposition puzzle. What value is there in analyzing the validity of implausible (or even impossible) hypotheses? Should we reinvestigate whether or not the world is flat? No, we don’t need to do that because previous inquiries as to its shape have been firmly and incontrovertibly resolved. As David Gorski puts it, “Wild inconsistencies with firmly established knowledge can in some cases be adequate for rejecting a hypothetical treatment as effective (homeopathy, for instance).”

So why are we spending any time on the “shape of the earth” type questions? I think it’s partially because Evidence Based Medicine (EBM) has been incorrectly positioned as the one and only analytical tool in the physician’s toolbox. I also suspect that our post-modernist culture encourages us to be silent about Emperors with no clothes. Of course, there are those who are making a handsome profit on “flat earth” memorabilia. But most of all, I believe it’s because we haven’t fully embraced the concept of Science Based Medicine (as opposed to EBM) as the foundation for appropriate scientific investigation.

Steven Novella explains:

EBM is a vital and positive influence on the practice of medicine, but it has its limitations… [especially] the focus on evidence to the exclusion of scientific plausibility…
All of science describes the same reality, and therefore it must (if it is functioning properly) all be mutually compatible. Collectively, science builds one cumulative model of the natural world. This means we can make rational judgments about what is likely to be true based upon what is already well established. This does not necessarily equate to rejecting new ideas out-of-hand, but rather to adjusting the threshold of evidence required to establish a new claim based upon the prior scientific plausibility of the new claim. Failure to do so leads to conclusions and recommendations that are not reliable, and therefore medical practices that are not reliably safe and effective.

The problem with EBM is that its original intent (as described by David Sackett) has been reduced in scope by popular opinion over time. It’s commonly held (and I’m simplifying here) that EBM means that objective evidence gathered in a randomized, controlled trial (RCT) is the only truly trustworthy means for determining cause and effect, relative efficacy, and mechanisms of action for treatment options.

Therefore it’s believed that incontrovertible conclusions cannot be drawn without a double blind, randomized, placebo-controlled trial. Obviously, it’s impossible to study every possible permutation of disease and treatment – so that relegates the majority of medicine to the “unproven” category. This does two things:

1. Third party payers love EBM because they can use it to deny treatment for things that have not yet been demonstrated to be effective in RCTs.

2. Pseudoscientists love EBM because it suggests that science is limited, and generally unable to offer conclusive evidence about most of its practices. This (they incorrectly believe) puts science and pseudoscience on an equal footing. Neither can be definitively proven effective in all cases, they reason, so they must be equally valid approaches to healing.

Let me give you a specific example of an actual conversation that I had with my favorite pseudoscientist, “Dr. John” (you might remember him from my first post). This is what he said to me one day:

There are no trials comparing drug X to drug Y in the setting of a patient on multiple other drugs with multiple other comorbidities… therefore you don’t really have EVIDENCE that drug X is appropriate or efficacious under those conditions. Since we don’t know if alternative therapies might be better in this case, there’s just as much reason to use alternative therapies as scientific therapies for this patient.

What happens here is that Dr. John has correctly pointed out that EBM has limitations (there wasn’t a 1:1 match up between the patient’s current circumstances and a clinical trial designed to assess the efficacy of drug X in those exact conditions), but then veers off into a non-sequitur conclusion: since we don’t have a clinical trial informing us regarding the exact best care of this patient, we should offer the patient any treatment we like.

What would Science Based Medicine say to Dr. John? It would say that in considering the best treatment option for the patient, we draw from a broad and deep scientific literature, the sum of which is more likely to help us solve the patients’ problems than the sum of testimonial anecdote. “Collectively, science builds one cumulative model of the natural world. This means we can make rational judgments about what is likely to be true based upon what is already well established.” Ironically, it is SBM that is truly holistic in its approach, not pseudoscience.

In closing I’d like to discuss one final important fallacy that contributes to the “more research is warranted” argument. There is a strong belief in Internet land that the “wisdom of crowds” (based on New Yorker columnist James Suroweicki’s book) can solve America ’s healthcare crisis. For example, if patients simply got together to share their collective wisdom about their treatment options – the treatment option with the highest rating would surely be the best one for that disease or condition, right? Then we wouldn’t need these narrow-minded, paternalistic doctors telling us what’s best for us. (In other words, more research is warranted, and that research should center upon personal interest and opinion).

Interestingly, in this scenario, narcotics and benzodiazepines rise to the top of the list for most conditions. Got back pain? Vicodin’s the best. Got fibromyalgia? Try Dilauded. Got cellulitis? Xanax. And so on and so forth. Apparently, being high is a great substitute for any medical treatment.

So where is the “wisdom” in all this? The argument stems from the idea that if a crowd of people guessed the number of jelly beans in a jar, the average of their guesses would be closer to the actual number than most individual guesses on their own.

However, the “wisdom of the crowds” is only as wise as the crowd being polled for the question at hand. If I asked a group of you readers about statins, I think I’d get a pretty reliable analysis of their pros, cons, side effect profiles and therapeutic values. Now, if I asked you to translate this sentence into ancient Sanskrit, I’m not sure that even your collective wisdom would suffice. Certainly the average of your attempts would not be more accurate than the one guy out there who could do it.

The wisdom of crowds argument is often used by pseudoscience proponents to justify further research into outdated and ineffective treatment options. You’ve heard this before: “Millions of Eastern peoples over millennia of using [insert favorite herb or treatment here] can’t be wrong!” Well, even modern masses of Americans believe that anxiolytics are great for treating infections – crowds can often be wrong.

Who can rescue us from this misinformation? What will stop the slow bleed of wasted research dollars on implausible therapies? Three simple words: science-based-medicine.

In conclusion: not all research warrants further investigation. Plausibility should be a precondition for medical investigations, and a holistic approach to analyzing the potential value of research is warranted. Evidence Based Medicine (as we commonly understand it) should be recognized as an excellent but limited tool. Science Based Medicine should be embraced as the new lens through which healthcare is evaluated. Only then can we move away from the foolish “wisdom of crowds” approach to offering anecdotally relevant treatment solutions to our patients. They deserve better.

Posted in: Clinical Trials, Science and Medicine

Leave a Comment (14) ↓

14 thoughts on “When Further Research Is NOT Warranted: The “Wisdom of Crowds” Fallacy

  1. Nick Barrowman says:

    For example, if patients simply got together to share their collective wisdom about their treatment options – the treatment option with the highest rating would surely be the best one for that disease or condition, right? … Interestingly, in this scenario, narcotics and benzodiazepines rise to the top of the list for most conditions.

    Seems plausible, but do you have a citation for this?

  2. suehiko says:

    This is a good entry. However, I would just like to digress a bit from what I take to be the point of this entry and add a few words on so called “Wise Crowds.” First, while it is true that pseudoscientists may use this concept to further their “arguments,” I believe there is some merit in and evidence for the Wise Crowd concept, and the baby should not be thrown out with the bath water.

    It is true that the jelly-beans-in-a-jar scenario is a rudimentary example of this phenomenon, but, the main premise of the wise crowd is that, UNDER THE RIGHT CONDITIONS, groups of people will, on average, out-perform (making predictions and making true statements of fact) the best performers within the group. Hence, it is true that “the ‘wisdom of the crowds’ is only as wise as the crowd being polled for the question at hand.”

    In conclusion, my only point is that the “Wisdom of Crowds Fallacy” is more of a misapplication.

  3. eldereft says:

    Every now and again we do actually get to see variants on the ‘more research is needed’ theme. In 2004 the US Department of Energy issued a report summarizing the state of the evidence for cold fusion, and how the field had evolved since the 1989 Pons and Fleischmann fiasco. They concluded that research along the lines of that which had been conducted for the preceding decade and a half was not promising, and future proposals should focus on new systems where below-stellar temperature fusion reactions might plausibly be observed, or should at least try not to suck so hard.

  4. wertys says:

    Interestingly I read a paper the other day which correlated sales of the commonest five supplements to the appearance of media reports about evidence of either benefit, lack of benefit or harm. the only thing which affected sales was evidence of harm, which in the case of Vitamin E caused a decrease in sales following publication and reporting of the evidence. The ‘Wisdom of Crowds’ seems to apply only to things that are perceived as risky. It seems people in general are fine with the idea that supplements can’t be shown to actually work.

    The reference is….
    Tilburt JC, Emanuel EJ, Miller FG.
    Does the evidence make a difference in consumer behavior? Sales of supplements before and after publication of negative research results.
    J Gen Intern Med. 2008 Sep;23(9):1495-8. Epub 2008 Jul 10…..

  5. adiemusfree says:

    There is an already functioning ‘wisdom of the crowd’ – Treatment Guidelines are often developed on the basis not just of ‘evidence’ from RCT’s but also from consensus of wise minds – that is, people with a scientific knowledge-base as well as knowledge of real clinical situations.
    A good example? In 1997, what is now called the Guidelines Group in New Zealand developed the Acute Low Back Pain Guidelines and at the same time published the first guide to the assessment of psychosocial yellow flags, or risk factors for chronicity.
    The algorithm or heuristic pathway was based not just on RCT-based evidence of what does and does not ‘work’ for acute low back pain, but also on pragmatic knowledge of people when they present to a general practitioner with acute low back pain.
    By consensus a set of workable recommendations were made, and the impact on treatment habits has been substantial – the words ‘psychosocial yellow flags’ at least is recognised internationally!
    The problem is when the ‘crowd’ in question is formed because of political input, unknowledgeable people with limited awareness of how science works, or from groups with a vested interest in the outcome (usually financial).

  6. yeahsurewhatever says:

    The internet, in democratizing knowledge, has led a lot of people to believe that it is also possible to democratize expertise. Couple that with the relative anonymity the internet provides, and the substrate is ripe for attention-seeking liars and financially-motivated fraud.

    I don’t actually think it’s science-based medicine that can rescue us from this, since it stems from a much larger problem that is not particular to medicine.

    There is so much misinformation and disinformation in the world today that many people fail to achieve basic cultural literacy about important things such as (but not limited to) medicine. They don’t even have a reliable starting point from which to proceed.

  7. yeahsurewhatever says:

    And of course suehiko’s argument is ridiculous, since invoking “THE RIGHT CONDITIONS” begs the question. We can only know if the conditions are right afterwards, by calibrating the results with those from the experts you claim you don’t need.

  8. Jules says:

    Sticking to models and building up from a strong knowledge base is great, but it’s the odd, curious, and seemingly implausible that leads to breakthroughs. A case in point: DNA. We know it as the blueprint of life (or whatever cliche floats your boat), but for a long time it was thought that proteins were the basis through which information was transmitted. Only after Watson and Crick’s definitive experiment (not their structure, but their model of replication) was the protein/DNA controversy finally laid to rest.

    If you think about it, the scientists who posited that proteins were the source of heredity actually made a more plausible case than those who suggested that DNA might be key. After all, life is complicated. Much too complicated for four (six if you count ribose and phosphate) relatively simple molecules to control–so the theory went.

    A clinical example: there are several studies claiming that epilepsy can be controlled by inducing metabolic ketosis (you can find them easily on PubMed). Is this plausible? Knowing what we know about epilepsy, the answer is a flat-out “No.” Is it true? Again, knowing what we know about epilepsy, the answer is “No.” Yet, the studies that have been done seem to suggest that it can, and does.

    It could be that those studies are defective in some way that renders their results useless. Or it could be that it does work, but bugger if we can figure out how based on our current understandings of how the brain works.

    While I understand and, to a large part, agree with the idea that we should investigate based on plausibility, let’s not forget that it is the willingness to investigate the implausible that leads us to the discoveries that change science and medicine.

  9. Jurjen S. says:

    Quoth “Dr. John”:

    Since we don’t know if alternative therapies might be better in this case, there’s just as much reason to use alternative therapies as scientific therapies for this patient.

    By that line of reasoning, “there’s just as much reason” to shoot the patient. After all, even though we have a large body of empirical evidence that people who are shot suffer massive trauma and often die as a result, “there is no evidence” that shooting this particular individual will lead to the same result, right? In his particular case, who knows, it might prove beneficial. “There is no evidence” that it won’t, provided you disregard the pre-existing body of knowledge.

  10. Val Jones says:

    Yeahsurewhatever hit the nail on the head:

    “The internet, in democratizing knowledge, has led a lot of people to believe that it is also possible to democratize expertise.”

    That’s precisely what I was getting at (although it took me a long post to get there). Of course I don’t disagree with suehiko’s view that crowds can provide wisdom under the RIGHT conditions. Most people in Internet land have missed that point, however.

    Jurjen S. has nicely summarized the logical conclusion of “Dr. John’s” line of reasoning: “Why not shoot the patient?” The Dr. Johns of the world are exasperating indeed.

    As for Jules – there is (in my opinion) a big difference in the discovery of DNA versus the continuing analysis of the potential value of homeopathy. Of course scientists are always looking for that eureka moment (when we discover a new piece of the phenomena puzzle) but will it come through the further study of a fully debunked and outdated system of thought? I’m sure it won’t. I suppose this all relates back to Mark Crislip’s explanation of the implausible versus the impossible. Another important distinction. http://www.sciencebasedmedicine.org/?p=204

    Thanks again for all your thoughtful contributions.

  11. Joe says:

    Jules on 24 Oct 2008 at 6:27 am wrote “let’s not forget that it is the willingness to investigate the implausible that leads us to the discoveries that change science and medicine.”

    I hate to wax philosophical; but I think implausibility has two extremes. There was no reason to think anything in yew (Taxus species) would be useful in cancer treatment. Random screening discovered Taxol. (One could note that it is reasonable to expect to find useful molecules in nature; but the specific activity was unexpected.) On the other extreme, there are reasons to think that homeopathy and reiki (for example) cannot work.

    The most insidious use of “more research is needed” is in the reports of studies funded by the National Center for Complementary and Alternative Medicine (NCCAM). It almost seems that phrase comes pre-printed on every report they generate. They never report failures- only “the results do not support the treatment, more research is needed.”

    Before anyone picks a nit; I think the NCCAM funded the 2005 report that concluded echinacia is not effective against colds; but they followed with a press release saying that the dose used in the study was probably inadequate. You can read about that sort of post hoc rationalization in James Randi’s book “Flim Flam.” It is a common ploy in woo.

  12. KarlS says:

    Agree with your points on plausibility, but suspect that the SBM term will be missused to promote herbal interventions based on cell culture experiments that are, arguably, scientific. If scientists are studying it, they will say, why wait for EBM to prove it conclusively: buy my herbs now!

    Perhaps Clinically Plausible SBM?

  13. I had to chuckle at this, because “Further research is warranted” was the key phrase all us undergrads learned to stick into our lab reports whenever our Organic Chem experiments epically failed! (Hmmmm… SCAM = Epic Fail? :))

Comments are closed.