Posts Tagged Bayesian analysis

P Value Under Fire

coinflipThe greatest strength of science is that it is self-critical. Scientists are not only critical of specific claims and the evidence for those claims, but they are critical of the process of science itself. That criticism is constructive – it is designed to make the process better, more efficient, and more reliable.

One aspect of the process of science that has received intense criticism in the last few years is an over-reliance on P-values, a specific statistical method for analyzing data. This may seem like a wonky technical point, but it actually cuts to the heart of science-based medicine. In a way the P-value is the focal point of much of what we advocate for at SBM.

Recently the American Statistical Association (ASA) put out a position paper in which they specifically warn against misuse of the P-value. This is the first time in their 177 years of existence they have felt the need to put out such a position paper. The reason for this unprecedented act was their feeling that abuse of the P-value is taking the practice of science off course, and a much needed course correction is overdue. (more…)

Posted in: Science and Medicine

Leave a Comment (0) →

Psychology Journal Bans Significance Testing

p-valuesThis is perhaps the first real crack in the wall for the almost-universal use of the null hypothesis significance testing procedure (NHSTP). The journal, Basic and Applied Social Psychology (BASP), has banned the use of NHSTP and related statistical procedures from their journal. They previously had stated that use of these statistical methods was no longer required but can be optional included. Now they have proceeded to a full ban.

The type of analysis being banned is often called a frequentist analysis, and we have been highly critical in the pages of SBM of overreliance on such methods. This is the iconic p-value where <0.05 is generally considered to be statistically significant.

The process of hypothesis testing and rigorous statistical methods for doing so were worked out in the 1920s. Ronald Fisher developed the statistical methods, while Jerzy Neyman and Egon Pearson developed the process of hypothesis testing. They certainly deserve a great deal of credit for their role in crafting modern scientific procedures and making them far more quantitative and rigorous.

However, the p-value was never meant to be the sole measure of whether or not a particular hypothesis is true. Rather it was meant only as a measure of whether or not the data should be taken seriously. Further, the p-value is widely misunderstood. The precise definition is:

The p value is the probability to obtain an effect equal to or more extreme than the one observed presuming the null hypothesis of no effect is true.


Posted in: Basic Science, Clinical Trials, Science and Medicine

Leave a Comment (117) →

Beware The P-Value

Part of the mission of SBM is to continually prod discussion and examination of the relationship between science and medicine, with special attention on those beliefs and movements within medicine that we feel run counter to science and good medical practice. Chief among them is so-called complementary and alternative medicine (CAM) – although proponents are constantly tweaking the branding, for convenience I will simply refer to it as CAM.

Within academia I have found that CAM is promoted largely below the radar, with the deliberate absence of public debate and discussion. I have been told this directly, and that the reason is to avoid controversy. This stance assumes that CAM is a good thing and that any controversy would be unjustified, perhaps the result of bigotry rather than reason. It’s sad to see how successful this campaign has been, even among my fellow academics and scientists who should know better.

The reality is that CAM is fatally flawed in both philosophy and practice, and the claims of CAM proponents wither under direct light. I take some small solace in the observation that CAM is starting to be the victim of its own success – growing awareness of CAM is shedding some inevitable light on what it actually is. Further, because CAM proponents are constantly trying to bend and even break the rules of science, this forces a close examination of what those rules should actually be, how they work, and their strengths and weaknesses.


Posted in: Clinical Trials

Leave a Comment (179) →

Of the Trial to Assess Chelation Therapy, Bayes, the NIH, and Human Studies Ethics

An experiment is ethical or not at its inception; it does not become ethical post hoc—ends do not justify means.
~ Henry K. Beecher


A couple of weeks ago, Dr. Josephine Briggs, the Director of the National Center for Complementary and Alternative Medicine (NCCAM), posted a short essay on the NCCAM Research Blog touting the results of the Trial to Assess Chelation Therapy (TACT) (italics added):

The authors found that those receiving the active treatment clearly fared better than those receiving placebo. The accompanying editorial in the AHJ reminds readers about the value of equipoise and the need to “test our beliefs against evidence.”

Most physicians did not expect benefit from chelation treatment for cardiovascular disease. I readily admit, initially, I also did not expect we would find evidence that these treatments reduce heart attack, strokes, or death. So, the evidence of benefit coming from analyses of the TACT trial has been a surprise to many of us. The subgroup analyses are suggesting sizable benefit for diabetic patients—and also, importantly, no benefit for the non-diabetic patient. Clearly subgroup analyses, even if prespecified, do not give us the final answer. But it is also clear that more research is needed to test these important findings.

And TACT findings are indeed a reminder of the importance of retaining equipoise [sic], seeking further research aimed at replicating the findings, and neither accepting nor rejecting findings based on personal biases. The scientific process is designed to weed out our preconceived notions and replace them with evidence.

Dr. Briggs concluded:

So, TACT is a reminder—an open mind is at the center of the scientific method.

Dr. Briggs’s title was “Bayes’ Rule and Being Ready To Change Our Minds”, a reference to a recent editorial that had accompanied one of the TACT papers. That editorial, by Dr. Sanjay Kaul, a physician and statistician from UCLA, begins with this quotation:

Preconceived notions are the locks on the door to wisdom.
~ Merry Browne

Here is the relevant passage from Dr. Kaul’s editorial (italics added):

Sixth, it has been argued that the trial was unethical because there was no compelling clinical or preclinical evidence that chelation therapy has significant efficacy against atherosclerotic cardiovascular disease, and given that chelation therapy can cause harm, the risk was not minimal. A Bayesian analysis would not look kindly on the results because of the low prior probability of treatment effect (the so-called implausibility argument).6 This is an uncharitable (and unwarranted) interpretation of the data because previous systematic reviews concluded, “insufficient evidence to decide on the effectiveness or ineffectiveness of chelation therapy in improving clinical outcomes among people with atherosclerotic cardiovascular disease.” It is axiomatic that absence of evidence of efficacy is not the same as evidence of the absence of efficacy.

From a Bayesian perspective, the strength of evidence is often summarized using a Bayes factor, which is a measure of how well 2 competing hypotheses (the null and the alternate) predict the data. The Bayes factor and the corresponding strength of evidence for the primary end point result in TACT overall, and diabetic cohorts are shown in Table 1. The p-value of 0.035 for TACT overall cohort translates into a Bayes factor of 0.108, which means the evidence supports the null hypothesis ≈1/9th as strongly as it does the alternative. This reduces the null probability from 50% pretrial (justified by suspension of one’s belief in treatment effect) to 10% post-trial. Although this does not represent strong evidence against the null, it does reduce the level of skepticism surrounding chelation therapy. In the diabetic cohort, the nominal p-value of 0.0002 translates into a Bayes factor of 0.002 (1/500), which reduces the extremely skeptical prior null probability of 95% to 4% post- trial, indicating very strong evidence against the null.

In concluding, Dr. Kaul states:

Finally, TACT highlights the double standard when it comes to accepting inconvenient results not aligned with our preconceived notions on so-called dubious quack cures such as chelation…

Closed minds?

Dr. Kaul’s reference “6” above is to a lengthy article that we published in 2008 titled “Why the NIH Trial to Assess Chelation Therapy Should Be Abandoned”. So, it seems, both Drs. Briggs and Kaul were chastising us for our biased, preconceived beliefs about so-called dubious quack cures. Our minds were, apparently, not open. Let’s examine this contention. (more…)

Posted in: Clinical Trials, Health Fraud, Medical Academia, Medical Ethics, Politics and Regulation

Leave a Comment (54) →

5 out of 4 Americans Do Not Understand Statistics

Ed: Doctors say he’s got a 50/50 chance at living.
Frank: Well there’s only a 10% chance of that
Naked Gun

There are several motivations for choosing a topic about which to write. One is to educate others about a topic about which I am expert. Another motivation is amusement; some posts I write solely for the glee I experience in deconstructing a particular piece of nonsense. Another motivation, and the one behind this entry, is to educate me.

I hope that the process of writing this entry will help me to better understand a topic with which I have always had difficulties: statistics. I took, and promptly dropped, statistics 4 times a college. Once they got past the bell shaped curve derived from flipping a coin I just could not wrap my head around the concepts presented. I think the odds are against me, but I am going to attempt, and likely fail, in discussing some aspects of statistics that I want to understand better. Or, as is more likely, learn for the umpteenth time, only to be forgotten or confused in the future. (more…)

Posted in: Basic Science, Science and Medicine

Leave a Comment (121) →

A Skeptical Look at Screening Tests

I’m going to follow Mark Crislip’s example and recycle my presentation from The Amazing Meeting last week, not because I’m lazy or short on time (although I am both), but because I think the information is worth sharing with a larger audience.

We’ve all had screening tests and we’re all likely to have more of them, but there is a lot of misinformation and misunderstanding about what screening tests can and can’t do. Screening tests are done on populations of asymptomatic people and must be distinguished from diagnostic tests done on individual patients who have symptoms. Some tests are excellent for diagnostic purposes but are not appropriate for screening purposes.

We’re constantly being admonished to get tested for one thing or another. A typical example was a recent Dear Abby column. She got a letter from a woman who had been screened for kidney disease and learned that she had a mild decrease in kidney function. Abby was shocked to learn that 26 million Americans have chronic kidney disease, and she advised her readers to get their kidneys checked. This was terrible advice. It superficially seems like good advice, because if you have something wrong with your kidneys, you’d want to know about it, right? In fact, if there was anything wrong anywhere in your body, you’d want to know about it. By that logic, it might seem advisable to test everyone for everything. But that would be stupid. It would find lots of false positives, it would create anxiety by picking up harmless variants and anomalies that never would have caused problems, it would be expensive, and it would do more harm than good.

Posted in: Cancer, Diagnostic tests & procedures

Leave a Comment (24) →

“Moneyball,” the 2012 election, and science- and evidence-based medicine

Regular readers of my other blog probably know that I’m into more than just science, skepticism, and promoting science-based medicine (SBM). I’m also into science fiction, computers, and baseball, not to mention politics (at least more than average). That’s why our recent election, coming as it did hot on the heels of the World Series in which my beloved Detroit Tigers utterly choked got me to thinking. Actually, it was more than just that. It was also an article that appeared a couple of weeks before the election in the New England Journal of Medicine entitled Moneyball and Medicine, by Christopher J. Phillips, PhD, Jeremy A. Greene, MD, PhD, and Scott H. Podolsky, MD. In it, they compare what they call “evidence-based” baseball to “evidence-based medicine,” something that is not as far-fetched as one might think.

“Moneyball,” as baseball fans know, refers to a book by Michael Lewis entitled Moneyball: The Art of Winning an Unfair Game. Published in 2003, Moneyball is the story of the Oakland Athletics and their manager Billy Beane and how the A’s managed to field a competitive team even though the organization was—shall we say?—”revenue challenged” compared to big market teams like the New York Yankees. The central premise of the book was that that the collective wisdom of baseball leaders, such as managers, coaches, scouts, owners, and general managers, was flawed and too subjective. Using rigorous statistical analysis, the A’s front office determined various metrics that were better predictors of offensive success than previously used indicators. For example, conventional wisdom at the time valued stolen bases, runs batted in, and batting average, but the A’s determined that on-base percentage and slugging percentage were better predictors, and cheaper to obtain on the free market, to boot. As a result, the 2002 Athletics, with a payroll of $41 million (the third lowest in baseball), were able to compete in the market against teams like the Yankees, which had a payroll of $125 million. The book also discussed the A’s farm system and how it determined which players were more likely to develop into solid major league players, as well as the history of sabermetric analysis, a term coined by one of its pioneers Bill James after SABR, the Society for American Baseball Research. Sabermetrics is basically concerned with determining the value of a player or team in current or past seasons and with predicting the value of a player or team in the future.

Posted in: Clinical Trials, Politics and Regulation, Science and Medicine, Science and the Media

Leave a Comment (47) →

The Power of Replication – Bems Psi Research

I love reading quotes by the likes of Karl Popper in the scientific literature. A recent replication of Bem’s infamous psi research, Feeling the Future, gives us this quote:

Popper (1959/2002) defined a scientifically true effect as that “which can be regularly reproduced by anyone who carries out the appropriate experiment in the way prescribed.”

The paper is the latest replication of Daryl Bem’s 2011 series of 9 experiments in which he claimed consistent evidence for a precognitive effect, or the ability of future events to influence the present. The studies were published in The Journal of Personality and Social Psychology, a prestigious psychology journal. All of the studies followed a similar format, reversing the usually direction of standard psychology experiments to determine if future events can affect past performance.

In the 9th study, for example, subjects were given a list of words in sequence on a computer screen. They were then asked to recall as many of the words as possible. Following that they were given two practice sessions with half of the word chosen by the computer at random. The results were then analyzed to see if practicing the words improved the subject’s recall for those words in the past. Bem found that they did, with the largest effect size of any of the 9 studies.


Posted in: Neuroscience/Mental Health, Science and Medicine

Leave a Comment (18) →

What is Science?

Consider these statements:

…there is an evidence base for biofield therapies. (citing the Cochrane Review of Touch Therapies)

The larger issue is what constitutes “pseudoscience” and what information is worthy of dissemination to the public. Should the data from our well conducted, rigorous, randomized controlled trial [of ‘biofield healing’] be dismissed because the mechanisms are unknown or because some scientists do not believe in the specific therapy?…Premature rejection of findings from rigorous randomized controlled trials are as big a threat to science as the continuation of falsehoods based on belief. Thus, as clinicians and scientists, our highest duty to patients should be to investigate promising solutions with high benefit/risk ratios, not to act as gatekeepers of information based on personal opinion.

–Jain et al, quoted here

Touch therapies may have a modest effect in pain relief. More studies on HT and Reiki in relieving pain are needed. More studies including children are also required to evaluate the effect of touch on children.

Touch Therapies are so-called as it is believed that the practitioners have touched the clients’ energy field.

It is believed this effect occurs by exerting energy to restore, energize, and balance the energy field disturbances using hands-on or hands-off techniques (Eden 1993). The underlying concept is that sickness and disease arise from imbalances in the vital energy field. However, the existence of the energy field of the human body has not been proven scientifically and thus the effect of such therapies, which are believed to exert an effect on one’s energy field, is controversial and lies in doubt.

—Cochrane Review of Touch Therapies, quoted here


Science is advanced by an open mind that seeks knowledge, while acknowledging its current limits. Science does not make assertions about what cannot be true, simply because evidence that it is true has not yet been generated. Science does not mistake absence of evidence for evidence of absence. Science itself is fluid.

—David Katz

When people became interested in alternative medicines, they asked me to help out at Harvard Medical School. I realized that in order to survive there, one had to become a scientist. So I became a scientist.

—Ted Kaptchuk, quoted here.

 …It seems that the decision concerning acceptance of evidence (either in medicine or religion) ultimately reflects the beliefs of the person that exist before all arguments and observation.

 —Ted Kaptchuk, quoted here.

Together they betray a misunderstanding of science that is common not only to “CAM” apologists, but to many academic medical researchers. Let me explain. (more…)

Posted in: Basic Science, Book & movie reviews, Clinical Trials, Energy Medicine, Faith Healing & Spirituality, Homeopathy, Medical Academia, Science and Medicine

Leave a Comment (37) →

On the “individualization” of treatments in “alternative medicine,” revisited

As I contemplated what I’d like to write about for the first post of 2012, I happened to come across a post by former regular and now occasional SBM contributor Peter Lipson entitled Another crack at medical cranks. In it, Dr. Lipson discusses one characteristic that allows medical cranks and quacks to attract patients, namely the ability to make patients feel wanted, cared for, and, often, happy. As I (and several of us at SBM) have said before, it’s not necessary to invoke magic, quackery, or pseudoscience in order to show empathy to patients and provide them with the “human touch” that forges a strong therapeutic relationship between physician and patient and maximizes placebo effects without deception. In the old days, this used to be called “bedside manner,” but in these days of capitation and crappy third party payor reimbursement it’s very difficult for physicians to take the time necessary to listen to patients and thereby build the bonds of trust and mutual respect that can augment the treatments that are prescribed. Unfortunately, because of this the quacks have been all too eager to leap into the breach.

One aspect of this tendency of medical cranks is to claim that they somehow “individualize” their treatment to the patient, as Peter points out:

There are a number of so-called holistic doctors in town who claim to practice “individualized” medicine. What this really means isn’t clear. My colleagues and I certainly individualize the treatment plans for all of our patients, using data gleaned from decades of scientific studies of large groups of patients. What “individualized” care seems to mean in this other context is “stuff I made up to make that patient feel more unique and special.”


Posted in: Basic Science, Clinical Trials

Leave a Comment (32) →
Page 1 of 2 12