Daryl Bem is a respected psychology researcher who decided to try his hand at parapsychology. Last year he published a series of studies in which he claimed evidence for precognition — for test subjects being influenced in their choices by future events. The studies were published in a peer-reviewed psychology journal, the Journal of Personality and Social Psychology. This created somewhat of a controversy, and was deemed by some to be a failure of peer-review.
While the study designs were clever (he simply reversed the direction of some standard psychology experiments, putting the influencing factor after the effect it was supposed to have), and the studies looked fine on paper, the research raised many red flags — particularly in Bem’s conclusions.
The episode has created the opportunity to debate some important aspects of the scientific literature. Eric-Jan Wagenmakers and others questioned the p-value approach to statistical analysis, arguing that it tends to over-call a positive result. They argue for a Bayesian analysis, and in their re-analysis of the Bem data they found the evidence for psi to be “weak to non-existent.” This is essentially the same approach to the data that we support as science-based medicine, and the Bem study is a good example of why. If the standard techniques are finding evidence for the impossible, then it is more likely that the techniques are flawed rather than the entire body of physical science is wrong.