Carl Sagan supposedly once said that randomness is clumpy. Those three words have become one of my favorite go-to quotes, particularly when teaching residents and medical students who are often overly impressed with improbable runs of similar diagnoses or exam findings. I love this quote because it is so simple and yet reveals so much about our experience with observing the natural world. Sagan’s ability to offer up insightful nuggets of rational thought, even if he didn’t actually produce this gem, was unmatched and his efforts to bring science and reason to the public have been sorely missed. If you haven’t read any of Sagan’s works, I highly recommend The Demon-Haunted World: Science as a Candle in the Dark.
If you have a coin, and a few hours to kill, record the results of a long run of flips and you’ll see what Sagan meant about the nature of randomness. You will inevitably observe clusters of heads or tails that might seem improbable, but eventually the outcomes will average out to about half of the flips being heads and half resulting in tails. The more trials that you perform, the closer the outcomes will approach 50% for each possible result, assuming you aren’t gaming the system by using a trick coin.
I don’t think that very many people would argue with the fact that on average a coin flip is random chance, although there are still people out there who think that the Earth is flat and that Justin Bieber is a reptilian humanoid. But because of a deeply rooted cognitive bias, the gambler’s fallacy, we frequently fail at recognizing that randomness is clumpy. We accept the established overall odds, but our acceptance wavers in the face of short runs that go against our expectations. This error in logic can lead to the belief, for instance, that after five heads in a row there is a higher than 50% chance that the next flip with land on tails as if to magically even things out.
In my line of work as a pediatric hospitalist, I frequently experience other healthcare professionals making this mistake in a variety of circumstances. There is a known likelihood of bacteremia when an infant less than 28 days of life is evaluated for fever, for example. Despite this, it is common for physicians and nurses to lament, upon seeing fever as the triage chief complaint, that they are due for this life threatening infection after a number of recent febrile neonates have had negative blood cultures. (more…)