Articles

Stop Making Sense

I usually rely on the Secret.  Every two weeks or so the Universe offers up some bit of wacky whimsey and I have a topic for an SBM blog entry.  This week the Universe has failed me. Nothing has crossed my LCD so I have no studies to evaluate and I have been unusually busy at work preventing my browsing the Interwebs for material.  But try telling that to the Managing Editor.   I write half to amuse myself, half to learn about the topic, and half to clarify in my own mind the topics at hand (1).  So this week  is content free idle thoughts for my own benefit.

I have been reading 13 Things That Don’t Make Sense by Michael Brooks. The book concerns topics in science that are unexplained by the current understanding  of the laws of the universe or contradict the dominant paradigm. Well, almost.  His final topic is homeopathy, and it is the one topic whose conclusions, while qualified, belong on Failblog.  The first chapter concerns dark matter and dark energy and how what we can see makes up only a small fraction of the content of the universe.  The author discusses the history behind the discovery of dark matter and dark energy, and the theories that attempt to explain the cosmological measurements that demonstrate the dark existence.  Besides new kinds of matter, physicists have hypothesized that the problem is not undiscovered matter but a lack of understanding of the nature of gravity.  He discusses at length the Pioneer anomaly, the fact that the Pioneer space probes, launched in the 1970’s, are not following their projected trajectories and perhaps this was a manifestation of the dark matter/dark energy/modified gravity.  Evidently it wasn’t, the perturbations are probably  due to radiation pressures.

What is striking is the description of the approach to the measurements that led to the hypothesis of dark energy and matter and the Pioneer anomaly. The investigators took extreme care in looking for every possible error within known physical parameters rather than looking for new physics to explain the data.  The lead investigator of the Pioneer anomaly said something to the effect, when you go into something looking for something new, that is what you will see.

I do not have an exact quote, since I am reading (2) the Audible book version, and the one problem with audio books is you cannot underline important quotes.

That is a striking characteristic of the evaluations of observations that contradict the known laws of the universe.  First, the researchers were meticulous at looking into every possible reason for the anomalous results within known laws.  And when they found anomalies, like the paths of stars circling galactic centers that suggesting the presence of dark matter, they carefully peat and repeat the measurements to confirm the findings before releasing the results.

It is the care with which physicist/astronomers and others in the ‘hard’ scientist evaluate the natural world that is impressive.  They consistently rely first on the basic principals that have centuries of careful observation and experiment to validate their approximate truth.  In the case of the Pioneer anomaly, there was no new physics to explain the trajectory, although it took 30 years to find the solution.  In the case of stars orbiting the galactic center and the accelerating expansion of the universe, something unexplained was occurring.  In both cases, a meticulous comparison to known reality was key to the understanding of the validity of the observed phenomena.  They relied on basic principals.

The author also notes what happens when investigators evaluate phenomena with pre-existing ideas as to what they should find.  I have always used the example of N-Rays, but Brooks uses Percival Lowell as an example.  Lowell was an astronomer who was committed to finding a civilization of Mars, and as a consequence saw and mapped a complex canal system on the planet.  The ability to see what you want to be there, rather than what is actually there, is one of the hallmarks of bad science, although in Lowell’s case he may have been inadvertently mapping the vessels of his own eye.  I initially wanted to write an entire entry on the topic of scientific paradolia, but besides N-rays and Mars canals, I could not come up with similar examples.  Freud? Perhaps.

The counter example,  the disasters that result from  ignoring basic principles, and of avoiding repetition and careful measurement was typified by Pons and Fleishman and the cold fusion debacle.  There are phenomena not yet dreamt of in our philosophy, but if you are going to discover results that violate the basic laws of the universe, it had best be measured carefully and consistently reproduced.  Cold fusion does not meet those criteria.

It is quite a contrast with the studies of SCAM modalities.  Most SCAMs have no basis in known physics, chemistry, physiology, anatomy, etc.  Acupuncture, homeopathy, reiki, chiropractic and their ilk are not based in reality but fantasy.  It is the concept of prior plausibility, or what I would call a reality bias (3), that makes any positive findings more likely to error and bias.  It is clear, particularly well delineated with acupuncture, that these modalities have no effect on objective endpoints and only minor effects on subjective endpoints.  Increasingly well designed studies reveal decreasing effects until excellent studies show no effect.  SCAMs  are the medical N- rays and Mars canals.  Yet in medicine, rather than noting that the studies are based in unreality and repeating careful studies that help tease out the biases and mistakes that make most SCAM trials appear to have effectiveness, they open Centers for integrative, alternative and complementary therapies.

It is odd, the blindness of the medical practice for rank nonsense.  Brooks describes the opprobrium that Pons, Fleishman and others received for their trumpeting of imaginary breakthroughs, although I suspect it was not what they did, but how they did it, bypassing peer review in favor of a press release, that led to the pariah status they now apparently have.  Researchers in quackery, rather than exile in France, start Clinics and Institutes.  While the collapse of the careers of Pons and Fleishman et. al as described by Brooks seems excessive for the ‘crime,’ the only consequence  in medicine for practicing magic is cash.

Outside of medicine careful research confirms or denies our understanding of reality.  In medicine, at least with SCAMs to judge from the Bravewell report, reality is ignored.

The last two chapters of the book cover placebo and homeopathy.  In the case of placebo, Brooks is a little sloppy at times at differentiating the difference between subjective and objective outcomes.  In the example of homeopathy he focuses on a few anomalous studies that were never reproduced and a long, and interesting, description of mysteries of water, to wonder about the unknown properties of H2o that could validate homeopathy, all the while admitting homeopathy is, by current understanding, total nonsense.  But as he consistently noted in the prior chapters, anomalous studies that can’t be reproduced and violate the basic laws of science turn out to be nonsense.  Then he ignores the same lessons in his conclusions about homeopathy: based on a bit of popularity of the crowd (how can millions be wrong?) and the fact that since there other anomalies have led to insights, homeopathy may indeed have  something to it.  I doubt it.

Understanding often advances with anomalous studies that are reproducible and where more mundane explanations consistent with known physical laws are excluded. As Asimov noted,  “The most exciting phrase to hear in science, the one that heralds new discoveries, is not “Eureka” but “That’s funny…”  SCAMs remain funny, but not in the way Asimov suggested  As they stand today,  the anomalies of SCAM are no more than error and bias.  If Ioannidis is correct about real medicine, and he is, then he is doubly correct for SCAM research. Instead of learning from past research,  marginally positive results of poorly designed studies are seen as proof of efficacy, despite no prior plausibility for and the opportunities for error and bias are enormous.   And then you charge the ill and vulnerable for the privilege of receiving your nonsense.

Astrology is not taught as a subset of astronomy for good reason.  There is no alternative aviation, complementary engineering or integrative fire fighting.  Medicine is different and it sometimes saddens me.

Notes
1) 3 out of 2 Americans do not understand statistics.

2) Yes, I am reading the book.  Unless you say the blind are feeling the book.

3) Foreshadowing of SBM to come.

Posted in: Book & movie reviews, Homeopathy, Science and Medicine

Leave a Comment (21) ↓

21 thoughts on “Stop Making Sense

  1. Thank you for this thought provoking, if poignant, article. The failure of medicine to reject cam has to be the product of multiple things, including the desire to do everything for your patients and the fact that most doctors probably aren’t that interested in science. Some just may not know better. CAM is so loud that it’s easy to fall for some of it. I believed that glycemic index mattered for a long time, solely because I heard so much about it. If it weren’t for Dr. Hall recommending “Heart 411″, I’d probably believe in the necessity of the LDL sub fractions (particle, etc.) (Incendently, I emailed Dr. Nissen abut that and he got right back to me, how cool is that.)

    It’s almost mob tactics, just be so loud and ubiquitous that people almost accept it by default. After all, Dr. Oz, the big bad heart surgeon tells us that we can diagnose any pathology with ear phrenology.

    On the side, thank you for mentioning the Pioneer spacecraft. I have a framed picture of the Pioneer plaque (anybody google it if you haven’t seen it), and find it so strange to think that one day we’ll all be gone, and this little craft will be out there in space with a couple of crude pictures on it describing where we used to live.

  2. cervantes says:

    One reason for the unsettling conclusion is that it’s not just 3 out of 2 Americans, but most physicians, who don’t understand statistics. I don’t have the reference handy, but BMJ has done a lot on this and it has been found that, for example, only a small minority of physicians understand Bayes Theorem. Most believe that if a test is 90% specific, and you test positive, there is a 90% chance you have the crud. This is very important because it is Bayesian reasoning that would lead us to be highly skeptical of conclusions with low prior probability, such as the efficacy of homeopathic remedies. If we don’t have an intuitive Bayesian view of the world, we are much more susceptible to being impressed by coincidence.

    Medical education needs to include much more on research methodology including statistics and general critical thinking. Right now most physicians don’t get that in any depth, unless they go on to an MPH or MSc of some sort. Why wouldn’t they be susceptible to SCAM? They don’t have the sort of education that will protect them from it.

  3. Ed Whitney says:

    One thing to mention is the ironical tone in much of Brooks’ writing in that book. I do not have the book in front of me, but there was one passage in the homeopathy chapter which describes a visit to a laboratory where the purported remedies are being prepared. The process of succussion involves shaking and striking on an elastic body, which in the particular lab is an old King James Bible held together by rubber bands. The remedies being succussed included stuff like Gog and Magog and frog spawn; and Brooks says something about the laboratory having bypassed Romeo and Juliet and gone straight to Macbeth.

    That dry British humor is present through the entire book. This should be borne in mind when evaluating the homeopathy chapter.

  4. Toiletman says:

    Actually, there is alternative engineering and just like it’s medicine counterpart, it does not work. Just think of all the stuff that is supposed to use “free energy”.

  5. BobbyG says:

    Re: “statistics,” I would commend to everyone this cool blog:

    http://skepticalscalpel.blogspot.com/

    In his “Labels” link column, “statistics” is one of the topical options.

    btw, I wrote about “Bayesian” considerations in the context of “fighting terrorism” nearly a decade ago:

    http://www.bgladd.com/Total_Information_Awareness/

  6. windriven says:

    “There is no alternative aviation…”

    I had the misfortune to fly Olympic Airways once. That came perilously close to alternative aviation.

  7. Jan Willem Nienhuys says:

    I initially wanted to write an entire entry on the topic of scientific paradolia, but besides N-rays and Mars canals, I could not come up with similar examples.

    Something similar to N-rays is the black light (1896) of Gustave Le Bon. It made Getman physicists initially ignore Becquerel’s discovery of radioactivity, because they thought it was some other kind of French nonsense.

    I did some investigations of the Gauquelin Mars Effect. Michel Gauquelin was a French psychologist who was systematically demolishing astrology with statistical tests. But then he hit on an effect that he thought was real. He had noted the position in the sky of different planets at the time of birth of people from various professions (military, artists, sports, criminals, priests, doctors,…). Out of one hundred combinations the combination Mars rising (or in the south) with sports had a most prominent association. He found other combinations too, but Mars and Sports were most striking. The effect was rather small. Rather than 17% like ‘ordinary people’, those special professions had Mars or Jupiter or whatever up to 22% in the ‘right position’.

    Well, if you try one hundred combinations (he had also two diferrent interpretations for ‘Mars rising’) then it is no surprise that one of them comes out significantly. He thought he had something: there you have an important source of pseudo-knowledge: whe the ressearcher falls in love with a hypothesis. One ex-astrologer (the Australian Geoffrey Dean) thinks that around the beginning of the 20th century it was quite common for parents to misreport the time of birth to the municipal registry so the newly bornm would have better stars.

    My explanation (backed up with a lot of countng) is that Gauquelin often found that the information he got from town hall registries was wrong or incomplete or at least not conform what was written in all kinds of yearbooks, membership lists, encyclopedias. He must have (is my conjecture) thrown out ‘unreliable’ data, but in biased fashion. If a birth time of some kind of ‘dubious datum’ yielded the ‘right’ position, he kept it (“Ha! the book must have been wrong!”), but if it yielded the wrong position, he rejected it (“There must be some kind of error, maybe this is another guy, let’s not consider this, one can’t be too careful”). Moreover he got in the habit of defining for each branch of sports where to put the distinction between ‘truly eminent’ and somewhat lesser, and he claimed that the effect was only present in the creme de la creme.

    He worked himself deeper and deeper into the whole thing, and committed suicide about the time he was found out to commit actual fraud (probably not because of that, but because of very personal matters).

    A more medical example might be Pauling.

    By the way, it’s Fleischmann (with sch and nn ) not Fleishman .

  8. annappaa says:

    Most SCAMs have no basis in known physics, chemistry, physiology, anatomy, etc. Acupuncture, homeopathy, reiki, chiropractic and their ilk are not based in reality but fantasy.

    I think it’s pretty weird that so many “alternative” modalities defy the laws of nature, yet their proponents often claim they are “natural.” They are anything but!

  9. pmoran says:

    There is no alternative aviation, complementary engineering or integrative fire fighting.

    We do have “alternative” automotive engineering, with many gadgets claimed to decrease fuel consumption.

    You will perhaps get “alternative” anything so long as certain conditions are met–

    1. If it doesn’t work there is usually no great immediate harm to the user — except to the hip pocket, which can still permit a caveat emptor approach. This applies to most use of “alternative” medicine but not to planes dropping from the sky or fires blazing out of control, or buildings collapsing.

    2. People can convince themselves it does work, — possible with personal fuel consumption and inevitable with most aspects of medicine.

    3. As a result of 2, and in the case of medicine also due to the likelihood of placebo responses, there is a strong contingent of believers, ever ready to create doubt in the minds of the public, politicians and the legislature.

    It is the ability to raisethose doubts that is the key. While there is doubt people will insist on the right to try it out. And they do so without the scientific commitment that we assume should be the prerequisite to any medical decision.

  10. mousethatroared says:

    - pmoran We do have “alternative” automotive engineering, with many gadgets claimed to decrease fuel consumption.”

    Actually, alot of gadgets do decrease fuel consumption…(a tire gadge is one of them) but my husband, the engineer who works in fuel systems, is very quick to point out that engineers are NOT scientists. :)

  11. @pmoran,

    We don’t even have to look at CAM, or things that are remotely ridiculous to all fall victim to what you wrote in your #2 point. Let’s consider PSA testing for prostate cancer in men.

    In medicine, we used to screen for prostate cancer with PSA (prostate specific antigen, it’s detectable in blood work, for anyone who doesn’t know) and DRE (digital rectal exam, which is the pointer finger in the butt, for anyone who doesn’t know.) The problem is that neither one of those techniques were good for prostate cancer screening, and slight elevations in PSA or abnormal DREs resulted in prostate biopsies which were typically either benign or positive for a cancer that both would kill you so slowly that you were more likely to die of other causes first. Of course sometimes they caught advanced cancers with this screening, but the evidence showed that too many people were getting unnecessary biopsies and/or prostate removals to justify the screening.

    The USPSTF reviewed all the evidence and came up with these guidelines for PSA:

    The USPSTF gave the guidance a D recommendation, which means there is moderate or high certainty that the service has no net benefit or that the harms outweigh the benefits, and the task force discourages use of the service.

    However: My father had a PSA done and it was 4.5, which is barely above normal. His previous screenings were all normal. Science would have told him initially to not get a PSA. However, science also would have told him that if he got a PSA, and it was slightly elevated, to wait a few months and then re-check it to see if it went back down, because PSA can be transiently elevated due to other inflammatory conditions. Given that his wife (my mother) just went into remission from multiple myeloma, he got a biopsy and sure enough, he had prostate cancer with a Gleason score of 7. He’s going to have the computer guided or whatever prostatectomy in a few weeks at one of the top cancer facilities in the USA.

    This definitely makes me feel convinced that prostate cancer screening works, because, well, it caught a reasonably advanced cancer in my otherwise asymptomatic father. The most damned thing of it all is that had his prostate biopsy came back normal it would have merely reinforced my previously held belief that screening isn’t that useful. While screening is not recommended, it’s acceptable to freely discuss prostate cancer screening with a patient, and explain to them that while it is not reliable, that if the patient wants it, then we are more than happy to oblige. Will my new anecdote change the way I practice? Will I be more likely to push a patient towards screening and possibly getting unnecessary procedures because I know, for a fact, that PSA screening can catch a prostate cancer, but at the same time know that even if it finds a cancer, it may be so slowly evolving that the patient will likely die of other causes first.

  12. pmoran says:

    I see what you mean, Skepticalhealth, but is there any real inconsistency? The guidelines refer to the “net benefit” of screening of whole populations. This does not preclude the possibility that some lives may be saved. It merely finds them to be unduly costly in terms of overinvestigation, overtreatment and unnecessary morbidity and mortality in others, and (possibly) also overall costs.

  13. BillyJoe says:

    Skepticalhealth,

    Pmoran hit the nail on the head.
    You are using an anecdote to try to prove PSA tests are worthwhile. You can’t do that. The guidelines are based on studies employing tens of thousands of men. Your single case simply cannot overturn all of that. You might throw it into the pot with the other ten thousand men, but I think you can see that the guidelines are not going to change as a result.
    If doctors are swayed by personal experiences such as this one, that is just evidence that at least some doctors are not conversant with SBM.

  14. @BillyJoe, nobody says that PSA tests aren’t worthwhile. They are, it’s just that they may not be a very efficient/effective screening tool for prostate cancer because of the likelihood of performing unnecessary procedures for something that may or may not kill you quickly. PSA is actually very sensitive – it’s elevated in 95% of men with prostate cancer. (Make sure you understand what sensitive means in this respect, because it does not mean that if its elevated you have a 95% chance of having prostate cancer!) Furthermore, if one does have prostate cancer and has a prostatectomy, PSA is used as a marker for cancer activity in the body (ie, we check PSA to monitor the recurrence of the cancer.)

    My example was not to say that “science is wrong,” and I never said that the guidelines were poorly written or shady. If you look back to what pmoran wrote, he was saying that one reason people believe strange things is because it’s easy to convince themselves of something based on personal experience. My story about PSA was an example of how it could be very easy to convince yourself (myself) of something, even thought I am aware that my story is the exception, rather than the rule.

    If you’re interested in the actual guidelines, it’s rather interesting:

    http://www.uspreventiveservicestaskforce.org/uspstf08/prostate/prostaters.pdf

  15. qetzal says:

    I initially wanted to write an entire entry on the topic of scientific paradolia, but besides N-rays and Mars canals, I could not come up with similar examples.

    The Benveniste affair? Polywater? Wikipedia’s article on Pathological Science includes all these and more.

  16. BillyJoe says:

    SkepticalHealth,

    Sorry, I misread your post. You were actually giving an example of how the personal experiences of a doctor could easily lead to him abandoning the recommendations. And, in fact, they do. I can remember reading comments to an on-line article on PSA testing. Nearly every commenter related a patient they had dealt with that persuaded them to use the test outside the guidelines.

    However, I didn’t mean to say that the PSA tests was useless, just that it is not recommended as a screening test for prostate cancer. The RACGP guidelines here in Australia are similar. Interestingly they conflict with the guidelines issued by USANZ (Urological Society Australia New Zealand) who recommend a test at 40 for all men and then follow up depending on the result (low normal – test every five years; high normal – test annually). One of them is not following the evidence and I suspect it is the specialist college.

  17. DugganSC says:

    Incidentally, I was able to make use of the bit on dark matter in a conversation last weekend. {gives a thumbs-up} Thank you, Science Based Medicine Blog!

Comments are closed.