When Further Research Is NOT Warranted: The “Wisdom of Crowds” Fallacy

Most scientific research studies have at least one thing in common: the conclusion section ends with, “further research is warranted.” I’d say it’s about as common as the “talk to your doctor” disclaimer in TV ads for pharmaceutical products. And in a way, they both serve the same purpose. They’re a “CYA” move.

What does “further research is warranted” mean in plain English? I think it can be roughly translated: “My research study is not of the size or scope to fully explain all the phenomena described in this article. Therefore, draw conclusions beyond the data and study methods at your own risk. And yeah, my work is important and cool – so people should study it further.”

Of course, the first two sentences are reasonable – we should always remember not to draw conclusions beyond the information provided by the data we’ve collected (even though that’s about as challenging as getting a beagle not to eat a table scrap in an empty room). The real problem is the third sentence. Is the research promising enough to require further investment? How are we to know if further research is indeed warranted? I would argue that it should not be based solely on the subjective opinions of the researchers nor the popularity of the research topic to the general public.

As my colleagues here at Science Based Medicine have already explained, plausibility is an often overlooked but critical piece of the value proposition puzzle. What value is there in analyzing the validity of implausible (or even impossible) hypotheses? Should we reinvestigate whether or not the world is flat? No, we don’t need to do that because previous inquiries as to its shape have been firmly and incontrovertibly resolved. As David Gorski puts it, “Wild inconsistencies with firmly established knowledge can in some cases be adequate for rejecting a hypothetical treatment as effective (homeopathy, for instance).”

So why are we spending any time on the “shape of the earth” type questions? I think it’s partially because Evidence Based Medicine (EBM) has been incorrectly positioned as the one and only analytical tool in the physician’s toolbox. I also suspect that our post-modernist culture encourages us to be silent about Emperors with no clothes. Of course, there are those who are making a handsome profit on “flat earth” memorabilia. But most of all, I believe it’s because we haven’t fully embraced the concept of Science Based Medicine (as opposed to EBM) as the foundation for appropriate scientific investigation.

Steven Novella explains:

EBM is a vital and positive influence on the practice of medicine, but it has its limitations… [especially] the focus on evidence to the exclusion of scientific plausibility…
All of science describes the same reality, and therefore it must (if it is functioning properly) all be mutually compatible. Collectively, science builds one cumulative model of the natural world. This means we can make rational judgments about what is likely to be true based upon what is already well established. This does not necessarily equate to rejecting new ideas out-of-hand, but rather to adjusting the threshold of evidence required to establish a new claim based upon the prior scientific plausibility of the new claim. Failure to do so leads to conclusions and recommendations that are not reliable, and therefore medical practices that are not reliably safe and effective.

The problem with EBM is that its original intent (as described by David Sackett) has been reduced in scope by popular opinion over time. It’s commonly held (and I’m simplifying here) that EBM means that objective evidence gathered in a randomized, controlled trial (RCT) is the only truly trustworthy means for determining cause and effect, relative efficacy, and mechanisms of action for treatment options.

Therefore it’s believed that incontrovertible conclusions cannot be drawn without a double blind, randomized, placebo-controlled trial. Obviously, it’s impossible to study every possible permutation of disease and treatment – so that relegates the majority of medicine to the “unproven” category. This does two things:

1. Third party payers love EBM because they can use it to deny treatment for things that have not yet been demonstrated to be effective in RCTs.

2. Pseudoscientists love EBM because it suggests that science is limited, and generally unable to offer conclusive evidence about most of its practices. This (they incorrectly believe) puts science and pseudoscience on an equal footing. Neither can be definitively proven effective in all cases, they reason, so they must be equally valid approaches to healing.

Let me give you a specific example of an actual conversation that I had with my favorite pseudoscientist, “Dr. John” (you might remember him from my first post). This is what he said to me one day:

There are no trials comparing drug X to drug Y in the setting of a patient on multiple other drugs with multiple other comorbidities… therefore you don’t really have EVIDENCE that drug X is appropriate or efficacious under those conditions. Since we don’t know if alternative therapies might be better in this case, there’s just as much reason to use alternative therapies as scientific therapies for this patient.

What happens here is that Dr. John has correctly pointed out that EBM has limitations (there wasn’t a 1:1 match up between the patient’s current circumstances and a clinical trial designed to assess the efficacy of drug X in those exact conditions), but then veers off into a non-sequitur conclusion: since we don’t have a clinical trial informing us regarding the exact best care of this patient, we should offer the patient any treatment we like.

What would Science Based Medicine say to Dr. John? It would say that in considering the best treatment option for the patient, we draw from a broad and deep scientific literature, the sum of which is more likely to help us solve the patients’ problems than the sum of testimonial anecdote. “Collectively, science builds one cumulative model of the natural world. This means we can make rational judgments about what is likely to be true based upon what is already well established.” Ironically, it is SBM that is truly holistic in its approach, not pseudoscience.

In closing I’d like to discuss one final important fallacy that contributes to the “more research is warranted” argument. There is a strong belief in Internet land that the “wisdom of crowds” (based on New Yorker columnist James Suroweicki’s book) can solve America ’s healthcare crisis. For example, if patients simply got together to share their collective wisdom about their treatment options – the treatment option with the highest rating would surely be the best one for that disease or condition, right? Then we wouldn’t need these narrow-minded, paternalistic doctors telling us what’s best for us. (In other words, more research is warranted, and that research should center upon personal interest and opinion).

Interestingly, in this scenario, narcotics and benzodiazepines rise to the top of the list for most conditions. Got back pain? Vicodin’s the best. Got fibromyalgia? Try Dilauded. Got cellulitis? Xanax. And so on and so forth. Apparently, being high is a great substitute for any medical treatment.

So where is the “wisdom” in all this? The argument stems from the idea that if a crowd of people guessed the number of jelly beans in a jar, the average of their guesses would be closer to the actual number than most individual guesses on their own.

However, the “wisdom of the crowds” is only as wise as the crowd being polled for the question at hand. If I asked a group of you readers about statins, I think I’d get a pretty reliable analysis of their pros, cons, side effect profiles and therapeutic values. Now, if I asked you to translate this sentence into ancient Sanskrit, I’m not sure that even your collective wisdom would suffice. Certainly the average of your attempts would not be more accurate than the one guy out there who could do it.

The wisdom of crowds argument is often used by pseudoscience proponents to justify further research into outdated and ineffective treatment options. You’ve heard this before: “Millions of Eastern peoples over millennia of using [insert favorite herb or treatment here] can’t be wrong!” Well, even modern masses of Americans believe that anxiolytics are great for treating infections – crowds can often be wrong.

Who can rescue us from this misinformation? What will stop the slow bleed of wasted research dollars on implausible therapies? Three simple words: science-based-medicine.

In conclusion: not all research warrants further investigation. Plausibility should be a precondition for medical investigations, and a holistic approach to analyzing the potential value of research is warranted. Evidence Based Medicine (as we commonly understand it) should be recognized as an excellent but limited tool. Science Based Medicine should be embraced as the new lens through which healthcare is evaluated. Only then can we move away from the foolish “wisdom of crowds” approach to offering anecdotally relevant treatment solutions to our patients. They deserve better.

Posted in: Clinical Trials, Science and Medicine

Leave a Comment (14) ↓