Changing the Rules of Evidence

My daughter, Julia, loves to play games and has a bit of a competitive streak. She can make any activity into a game and is adept at making up rules on the spot. When she was younger, like most children, she had a tendency to add to or change the rules on the fly – usually to ensure a favorable outcome for herself. “Oh, Daddy, I forgot to mention that the ball can bounce once and that still counts.”

It was an opportunity for me to gently teach her that in order for rules to work everyone has to know what they are ahead of time and you can’t change them after the fact. Her smile told me that even at five she intuitively knew this already – that changing or making up new rules was not fair. What I was really teaching her was that she wasn’t going to get away with it with me, and by extension that it is socially unacceptable to mess with the rules to suit oneself.

Adults are really no different than children in our basic emotional makeup. We all want to change the rules to suit our own needs. The true difference is that as we mature we become more socially sophisticated; we become more subtle in our manipulations, and we develop the capacity to rationalize our wants and desires. We also learn that we are playing a bigger game – the social game. So we adhere to the rules of fairness, even if it means losing a competition, because we want to succeed at the more important game of socialization. (I’m not making any moral or ethical judgments here, just observing human behavior.)

Rules don’t just exist for games. Important systems have rules also, often ones that have evolved and developed over time to serve a particular purpose. For example, our democracy has rather complex rules. Political parties have their own rules for how they choose their nominee for president. Right now Hillary Clinton is desperately trying to change those rules in light of the results that have already emerged. She wants the popular vote to have the most importance, including those states (Florida and Michigan) who were deprived of their delegates for breaking the rules. Does anyone doubt for a moment that she is favoring this interpretation because it is the one favorable to her? Does anyone think that a year ago Senator Clinton would have argued that the popular vote was more important than the delegate count? This is nothing more than a more sophisticated version of my daughter’s flexible ball-game rules.

Science also has rules – very important rules. The goal of science is to develop models of the natural world that are reliable, that make predictions about future observations and experiments that turn out to be true. When we spend millions of dollars to launch a probe to Saturn, we want to be rewarded with beautiful pictures of the ringed planet. When we administer a medical treatment we want to be confident that we are more likely to help our patient than to harm them. With science, the stakes are high, so the rules are important.

Mostly the rules of science developed in order to limit our tendency to fool ourselves. That’s right – humans are remarkably adept at self-deception. Our perceptions are biased by our beliefs and desires. We are prone to confirmation bias, which means we notice and remember information that confirms what we already believe, and ignore or rationalize away disconfirming evidence. Our memories are based mostly on emotion – not fidelity of detail. Our intuitive logic is often flawed or oversimplified.

Science is simply a set of rules of investigation. The biggest rule of science is that we have to test our ideas against reality. We can’t just make stuff up and then assume we are right – we have to subject our guesses to observations that have the potential of proving them wrong. Scientific observations must be recorded objectively so that we don’t have to rely upon flawed memories. Outcomes need to be specified ahead of time – we cannot decide at the end of an experiment which results prove our hypothesis. Outcomes should be quantified as much as possible, and as objectively and unambiguously as possible.

Over time the rules of science became more rigorous. We decided that those judging the outcome of an experiment should be blinded, so that their biases could not influence their measurements. In medical studies we decided that double-blind studies were better, because when subjects know they are getting a real treatment they may report their outcomes differently or behave differently. Statistical methods developed to analyze the data mathematically, and cutoffs for significance became generally accepted.

The rules of science are still evolving – they are not written in stone. Also it is often challenging to figure out how to apply the rules of science to new types of questions. But there are many general principles of scientific investigation that hold in any context, and in general the more rigorously the rules of science are followed the more reliable the results.

Today there is a political/ideological movement within medicine and health care to change the rules after the fact. The purveyors of many sectarian methods of treatment and unscientific belief systems of health and illness have not succeeded at the fair rules of science. So now they want to change those rules. They want anecdotes to not only count but to trump rigorously controlled observations (that is, when the anecdotes are in their favor). They was to reinterpret the placebo effect after the fact as if it were a real effect. They want to count only those experiments that confirm their beliefs and ignore or reject those studies that reject their beliefs.

Being educated adults they have much more sophisticated language to express their childish desire to alter the rules.

Andrew Weil wants to relabel anecdotes he favors as “uncontrolled clinical observations.” This is a way of getting to choose after the fact which observations count, rather than letting the rules of science decide.

Dr. David Katz from Yale’s “Integrative Medicine” Program wants to allow for “a more fluid concept of evidence.” This way modalities he favors, such as homeopathy, that have failed by the generally accepted rules of science can still win with his more “fluid” rules.

When studies of “alternative” modalities are negative, proponents want to change the rules after they see the results. They claim that the “sham” acupuncture was giving a real effect too, or that the numbers in the study were too small, or that homeopathy cannot be tested with the same methods as cookie cutter drugs, or that a statistically insignificant trend in their favor should count even though the rules say they shouldn’t. Of course, when the outcome is positive, then these same rules are just fine. Heads I win, tales you lose.

When Congress created the National Center for Complementary and Alternative Medicine (NCCAM) they wanted to create a funding center within the NIH so that researchers who failed to get funding for their research through the existing rigorous rules could succeed with more relaxed rules made especially for them.

Many states in the US now have “healthcare freedom” laws. These laws essentially say that when practitioners fail to abide by the rules of ethical scientific medicine, it’s OK. They can have their own set of “alternative” rules in which they can do whatever they want without having to worry about that pesky “standard of care” or basing their diagnoses and treatments on tiresome “scientific evidence.”

As my daughter intuitively understood at age five – the rules only mean anything if everyone knows the rules ahead of time, if everyone has to play by the same rules, and if the rules cannot be changed ad hoc to favor one side.

The rules of science and evidence exist for a reason – they maximize the reliability of the conclusions we reach. In medicine, an applied science, this has immediate implications for the health of the public. We cannot have two systems of rules in medicine, one for”mainstream” medicine, and a second set of “fluid” rules for everything that fails to meet these rigorous mainstream rules. That is equivalent to having no rules at all. In fact it is worse, because it gives the false impression that there are rules, and this engenders a faith in the system that is not justified.

Unless there is one set of rules the entire system fails. We should not acquiesce to those who want to let the ball bounce once and still count.

Posted in: Clinical Trials, Science and Medicine

Leave a Comment (24) ↓

24 thoughts on “Changing the Rules of Evidence

  1. Calli Arcale says:

    “When we spend millions of dollars to launch a probe to Saturn, we want to be rewarded with beautiful pictures of the ringed planet.”

    FYI, there is some sensitivity among space geeks to the notion that the purpose of deep space probes is to take pretty pictures. This is in part because of pseudoscientific rubbish such as those claiming to have found evidence of alien civilizations in said pictures — or evidence of the photos being doctored to remove such evidence. Also, some of the “NASA is hiding stuff!” folks often cry foul with specious claims of the photos not being “true color”. (Seriously, they’re not, though NASA and ESA will occasionally release images processed to approximate true color.) They will argue that the mission is only worthwhile (and honest) if it can produce true-color images without any processing. (They are all highly suspicious of image processing, considering it tantamount to forgery.) Yet anyone who really follows these missions knows that true color would be *less* useful than what the probes are really returning. Never mind the technical challenges (and the philosophical debate about what “true color” actually means), you can get more science out of the pictures if they cover a larger part of the spectrum than the human eye is capable of.

    Plus, these probes do *so* much more than just return imagery. They are bristling with instrumentation. Multiple cameras, synthetic aperture radar, plasma wave instruments, particle counters, spectroscopes . . . even the spacecraft’s communications system is often used for scientific purposes, by observing changes in the signal as it passes behind other objects. The probe’s trajectory is even scientific, because it allows direct measurement of the masses of planets and moons, which in turn allows computation of average density. Cassini’s pictures have revealed a lot about Saturn. But they are only part of the entire package, and sometimes the most exciting scientific information comes from the non-imaging instruments.

    Still, it’s always the images that capture the public interest. Often to the exclusion of other data, which can be a problem. Some people rail against the idea of spending millions just to get pretty pictures of some lifeless rock that we’ll never visit in person. :-(

    That said, this article makes a very good and very true point: science is a set of rules that keep us honest, with others and, perhaps more importantly, with ourselves. They cannot be discarded merely because we dearly want our pet notion to be proven right.

  2. Harriet Hall says:

    I am reminded of a study I read that concluded, “X works, but this study was unable to demonstrate that fact.” :-)

  3. David Gorski says:

    That characterization would apply to most CAM studies and, unfortunately, more than a few “conventional” medicine studies.

  4. daijiyobu says:

    I had Katz as an instructor around 2000 when working on an ND at the University of Bridgeport

    — that I ceased for reasons I’ve stated elsewhere (see , ) —

    for a course held within Yale’s medical school, “Epidemiology and Public Health.”

    The UB (& partner Yale, at the time) ‘epistemic fraud’ that I experienced then continues to amaze me now, and it continues to amaze me that such is still going on.

    Katz’s recent language is in line my experience with ND unethical sectarian pseudoscience.

  5. apteryx says:

    Many MDs are noted to favor interventionism – testing, prophylaxis, “aggressive treatment” – in many different health contexts when there is no solid scientific evidence in favor of what they are doing, or even studies that definitely show outcomes are not improved. This certainly does not follow the rules of science, if medicine is treated strictly as an offshoot of science. However, I think it is a bit unfair to call it “childish.” 99% of all humans, including mature adults, are inclined to believe what they have been taught. If that’s a problem, it’s a failing of the whole species, and name-calling will do little to resolve it.

  6. Joe says:

    apteryx on 21 May 2008 at 3:37 pm wrote “Many MDs are noted to … when there is no solid scientific evidence in favor of what they are doing …”

    I await your justification of that statement with bated breath. Not to mention the rest of your statements … Ooops, I did mention them. For example, “99% of all humans …” ? Never mind, I won’t really hold my breath. Dream on about the “Apteryx Supremacy.” Do you think Matt Damon will play your role? Maybe Rob Schneider would be more appropriate.

  7. PalMD says:

    I do so enjoy these posts. It’s very hard to explain sometimes that wishing things were different doesn’t make them so.

  8. Harriet Hall says:

    “Many MDs are noted to favor interventionism – testing, prophylaxis, “aggressive treatment” – in many different health contexts when there is no solid scientific evidence in favor of what they are doing, or even studies that definitely show outcomes are not improved.”

    Unfortunately, there’s a lot of truth to that. Science goes up against the human desire to “do something.” One of the purposes of this blog is to make people more aware of our human failings and to promote more judicious application of rigorous scientific standards. I will be writing posts about over-diagnosis and over-treatment in the near future.

  9. ellazimm says:

    Excellent post Dr Steve. I’m going to send a link to some of my friends who need it.

  10. DLC says:

    Sometimes the need to “do something” is confabulated with the desire to be seen to be doing something. This is how we get bad laws on the books. Not being a medical professional myself, I am not qualified to say much on the subject on when a given intervention is called for. However, I do know that say, when I have a low-grade fever, muscle aches, stopped up head and incipient nausea or feverish feeling, taking some tylenol and drinking plenty of liquids will more often than not be the proper course of treatment. Of course, one time in who-knows-how-many times it may be the onset of pulmonary anthrax. . . but most of the time it’s just the flu.
    On the other hand, perhaps I should get my Qi adjusted or drink down some homeopathic water. . .
    Excuse me while I go peel the tape off my feet. :-)

  11. One important distinction that is critical to make:

    The mainstream medical community is philosophically science-based / evidence-based. There is an evolving standard of care. All practitioners fall short of the ideal standard to some degree. Some are brilliant, most are competent, and some are terrible. As Harriet wrote, one of our goals is to raise the overall standard and compliance with that standard.

    CAM, however, is an entirely different story. Part of the point of this article is that CAM is primarily about removing the standard of care and eliminating the scientific basis of medicine. CAM proponents want to change the institutions of medicine – they want to change the rules for their benefit.

    These are therefore two completely different problems – on the one hand falling short of the standard of care, and on the other deliberately eliminating it. Do not confuse the two.

  12. daijiyobu says:

    S.N. commented:

    “part of the point of this article is that CAM is primarily about removing the standard of care and eliminating the scientific basis of medicine.”

    I agree entirely, except I might alter the comment this way, just because this is what I observe:

    ‘CAM is about applying a scientific label to what is profoundly nonscientific.’

    In other words, CAM is essentially about ‘knowledge type fraud,’ and I’ll provide an example:

    how about labeling the supernatural spiritistic (see ),

    and the vitalistic teleological (see ),

    scientific (see's_Nat.htm ).

    A query: does anyone in Connecticut have any interest in an anonymous-like protest of this irrationality? (Ha!).

  13. daijiyobu says:

    Made a mistake in the previous comment post, kind of,

    UB has changed their ND Dean’s page regarding the spiritistic,

    so I refer you to this archived page, .


  14. SC says:

    I completely agree with your general point concerning CAM and its self-interested attempts at rule-revision.

    I do, though, have a few points to make:

    First, it’s kind of a shame that your examples of people attempting to change the rules mid-game are female. I’m sure it wasn’t intentional, but this does play into the stereotype of women’s being less appreciative of rules than men. This came up a lot, I recall, at the time of the Nykesha Sales deal at UConn a while back:

    While many male athletes at the time reported that similar rule-bending had long occurred in men’s sports (in addition, of course, to more serious violations of rules like the widespread use of artificial performance enhancers), this was used by many as an example of women’s emotionalism and inability to appreciate the value of abstract, non-case-sensitive rules.

    Which brings me to my second point: It may be true that women and other lower-status groups are more likely to think around established rules than men. If so, one possible reason for this is that rule systems are often designed to benefit the already-privileged. For any CAMers reading this, I am not here speaking of the rules of scientific inquiry discussed in Dr. Novella’s article, which as he has established are necessary, justified, and reasonable. However, if, say, the rules related to the process of applying for and receiving research funding favor certain privileged groups, I would say that excluded groups have a positive role to play in questioning which of those rules are detrimental to innovation and multiple perspectives and, in the case of those rules that are fully justified but at the same time exclude certain classes of people, thinking about how we can change things to allow more people to abide by them and still be successful.

    Given this, third, I want children to question rules and evaluate them critically, seeking to determine whether they are socially beneficial or justifiable. I also want them to think of themselves as people with the potential to participate in making, challenging, and reformulating rules. I believe these abilities develop through practice. So my approach to rules in the situation you described with your daughter would be slightly different. I would ask her to explain how she thinks that particular rule change would make the game better. I would also allow the change, on the condition that the game was still being played and that the rule would subsequently apply to all of the players and could only be changed with good justification.

  15. Skip says:

    I worry about what rules, if any, I will encounter this fall at an Osteopathic Medical School

  16. SC – for the record, I mentioned four people by name, two women, and two men, and one of the women was my daughter. I have two daughters, no sons, and she was used as an example of a child.

    Regarding changing the rules – I agree that children should be taught to question authority and that logic and facts trump authority. But this has to be balanced with fairness. The reason the rules of a game cannot be changed mid-stream is because it is not fair, regardless of whether or not the rules are better. If you want to improve the rules, then we start over with everyone knowing the new rules ahead of time.

  17. SC says:

    Ah! I was just directed here from Respectful Insolence and saw that you had replied to my comment several weeks ago!

    You did indeed mention two men and two women. Evidently, I was being hypersensitive. The sexist commentary surrounding the Sales record still rankles, for some reason.

    With regard to the invented-game scenario, I agree that if the rules are to be changed, the game would have to begin again from scratch in order for it to be fair. But I would encourage this challenge-justification-change-restart process. I dislike the idea of children growing up with an uncritical “rules are rules” attitude (not that I think you’re promoting this – just thought it should be emphasized).

    Thank you for your response.

Comments are closed.