Articles

Changing the Rules of Evidence

My daughter, Julia, loves to play games and has a bit of a competitive streak. She can make any activity into a game and is adept at making up rules on the spot. When she was younger, like most children, she had a tendency to add to or change the rules on the fly – usually to ensure a favorable outcome for herself. “Oh, Daddy, I forgot to mention that the ball can bounce once and that still counts.”

It was an opportunity for me to gently teach her that in order for rules to work everyone has to know what they are ahead of time and you can’t change them after the fact. Her smile told me that even at five she intuitively knew this already – that changing or making up new rules was not fair. What I was really teaching her was that she wasn’t going to get away with it with me, and by extension that it is socially unacceptable to mess with the rules to suit oneself.

Adults are really no different than children in our basic emotional makeup. We all want to change the rules to suit our own needs. The true difference is that as we mature we become more socially sophisticated; we become more subtle in our manipulations, and we develop the capacity to rationalize our wants and desires. We also learn that we are playing a bigger game – the social game. So we adhere to the rules of fairness, even if it means losing a competition, because we want to succeed at the more important game of socialization. (I’m not making any moral or ethical judgments here, just observing human behavior.)


Rules don’t just exist for games. Important systems have rules also, often ones that have evolved and developed over time to serve a particular purpose. For example, our democracy has rather complex rules. Political parties have their own rules for how they choose their nominee for president. Right now Hillary Clinton is desperately trying to change those rules in light of the results that have already emerged. She wants the popular vote to have the most importance, including those states (Florida and Michigan) who were deprived of their delegates for breaking the rules. Does anyone doubt for a moment that she is favoring this interpretation because it is the one favorable to her? Does anyone think that a year ago Senator Clinton would have argued that the popular vote was more important than the delegate count? This is nothing more than a more sophisticated version of my daughter’s flexible ball-game rules.

Science also has rules – very important rules. The goal of science is to develop models of the natural world that are reliable, that make predictions about future observations and experiments that turn out to be true. When we spend millions of dollars to launch a probe to Saturn, we want to be rewarded with beautiful pictures of the ringed planet. When we administer a medical treatment we want to be confident that we are more likely to help our patient than to harm them. With science, the stakes are high, so the rules are important.

Mostly the rules of science developed in order to limit our tendency to fool ourselves. That’s right – humans are remarkably adept at self-deception. Our perceptions are biased by our beliefs and desires. We are prone to confirmation bias, which means we notice and remember information that confirms what we already believe, and ignore or rationalize away disconfirming evidence. Our memories are based mostly on emotion – not fidelity of detail. Our intuitive logic is often flawed or oversimplified.

Science is simply a set of rules of investigation. The biggest rule of science is that we have to test our ideas against reality. We can’t just make stuff up and then assume we are right – we have to subject our guesses to observations that have the potential of proving them wrong. Scientific observations must be recorded objectively so that we don’t have to rely upon flawed memories. Outcomes need to be specified ahead of time – we cannot decide at the end of an experiment which results prove our hypothesis. Outcomes should be quantified as much as possible, and as objectively and unambiguously as possible.

Over time the rules of science became more rigorous. We decided that those judging the outcome of an experiment should be blinded, so that their biases could not influence their measurements. In medical studies we decided that double-blind studies were better, because when subjects know they are getting a real treatment they may report their outcomes differently or behave differently. Statistical methods developed to analyze the data mathematically, and cutoffs for significance became generally accepted.

The rules of science are still evolving – they are not written in stone. Also it is often challenging to figure out how to apply the rules of science to new types of questions. But there are many general principles of scientific investigation that hold in any context, and in general the more rigorously the rules of science are followed the more reliable the results.

Today there is a political/ideological movement within medicine and health care to change the rules after the fact. The purveyors of many sectarian methods of treatment and unscientific belief systems of health and illness have not succeeded at the fair rules of science. So now they want to change those rules. They want anecdotes to not only count but to trump rigorously controlled observations (that is, when the anecdotes are in their favor). They was to reinterpret the placebo effect after the fact as if it were a real effect. They want to count only those experiments that confirm their beliefs and ignore or reject those studies that reject their beliefs.

Being educated adults they have much more sophisticated language to express their childish desire to alter the rules.

Andrew Weil wants to relabel anecdotes he favors as “uncontrolled clinical observations.” This is a way of getting to choose after the fact which observations count, rather than letting the rules of science decide.

Dr. David Katz from Yale’s “Integrative Medicine” Program wants to allow for “a more fluid concept of evidence.” This way modalities he favors, such as homeopathy, that have failed by the generally accepted rules of science can still win with his more “fluid” rules.

When studies of “alternative” modalities are negative, proponents want to change the rules after they see the results. They claim that the “sham” acupuncture was giving a real effect too, or that the numbers in the study were too small, or that homeopathy cannot be tested with the same methods as cookie cutter drugs, or that a statistically insignificant trend in their favor should count even though the rules say they shouldn’t. Of course, when the outcome is positive, then these same rules are just fine. Heads I win, tales you lose.

When Congress created the National Center for Complementary and Alternative Medicine (NCCAM) they wanted to create a funding center within the NIH so that researchers who failed to get funding for their research through the existing rigorous rules could succeed with more relaxed rules made especially for them.

Many states in the US now have “healthcare freedom” laws. These laws essentially say that when practitioners fail to abide by the rules of ethical scientific medicine, it’s OK. They can have their own set of “alternative” rules in which they can do whatever they want without having to worry about that pesky “standard of care” or basing their diagnoses and treatments on tiresome “scientific evidence.”

As my daughter intuitively understood at age five – the rules only mean anything if everyone knows the rules ahead of time, if everyone has to play by the same rules, and if the rules cannot be changed ad hoc to favor one side.

The rules of science and evidence exist for a reason – they maximize the reliability of the conclusions we reach. In medicine, an applied science, this has immediate implications for the health of the public. We cannot have two systems of rules in medicine, one for”mainstream” medicine, and a second set of “fluid” rules for everything that fails to meet these rigorous mainstream rules. That is equivalent to having no rules at all. In fact it is worse, because it gives the false impression that there are rules, and this engenders a faith in the system that is not justified.

Unless there is one set of rules the entire system fails. We should not acquiesce to those who want to let the ball bounce once and still count.

Posted in: Clinical Trials, Science and Medicine

Leave a Comment (24) ↓