Most scientific research studies have at least one thing in common: the conclusion section ends with, “further research is warranted.” I’d say it’s about as common as the “talk to your doctor” disclaimer in TV ads for pharmaceutical products. And in a way, they both serve the same purpose. They’re a “CYA” move.
What does “further research is warranted” mean in plain English? I think it can be roughly translated: “My research study is not of the size or scope to fully explain all the phenomena described in this article. Therefore, draw conclusions beyond the data and study methods at your own risk. And yeah, my work is important and cool – so people should study it further.”
Of course, the first two sentences are reasonable – we should always remember not to draw conclusions beyond the information provided by the data we’ve collected (even though that’s about as challenging as getting a beagle not to eat a table scrap in an empty room). The real problem is the third sentence. Is the research promising enough to require further investment? How are we to know if further research is indeed warranted? I would argue that it should not be based solely on the subjective opinions of the researchers nor the popularity of the research topic to the general public. (more…)
The science in science-based medicine includes all of science, but relies most heavily on the biomedical literature – published studies that collectively represent our scientific medical knowledge. The scientific basis of medicine is only as good as this body of knowledge and the manner in which it is interpreted and put into practice.
We often discuss on this blog how to evaluate individual studies- the need for blinding, randomization, the importance of study size to meaningful statistical analysis, and other features that distinguish a reliable study from a worthless one. This is important, but only half of the equation. We also at times discuss the medical literature as it relates to a specific medical question or set of related questions – does homeopathy work or are statins beneficial for cholesterol reduction, for example. This requires not only the ability to judge individual studies, but a higher order analysis of the overall pattern of evidence among all relevant studies. Failure to do this, by focusing only on individual studies, results in the failure to see the forest for the trees.
It is this higher order analysis that I wish to discuss in this entry.
I have more thoughts on the homeopathy matter than fit in follow-up notes, so here goes.
First, David Gorski recalls the 1994 Pediatrics report on childhood diarrhea treated with tailored homeopathic remedies for each patient. There is more to the story than has been written. I am certain much of this will get back to the authors, but others may benefit from knowing how this group of homeopaths operate.
I recall the paper well, because it was the first journal report that I deconstructed and published (Pediatrics, Oct 1995) as a regular article. I think it was the first time the journal had published a formal rebuttal outside the Letters section. The head of pediatric pharmacy at Valley Medical Center, San Jose, brought the paper to me and asked what I thought if it. Bill London of National Council against Health Fraud and I spent six months discussing it and working over the details.
The paper had so many flaws, that one letter could not contain them. It had five or six end points, several arithmetical errors, graphs with missing data, only one end point reached consensus signficance (barely.)
Each case received a remedy tailored to the age, condition, duration of symptoms, appearance and odor of the stool, the recall of the parent or relative about stool frequency (which depended on how often the child’s diapers were changed, and a number of other qualities, typical of a homeopathic approach to diagnosis. The remedies given were not based on etiology, but based on personal observations, We saw that the remedy was chosen at a snapshot in time, depending on all those factors which varied from hour to hour. So the remedies depended on the time at which the child was brought in for examination and were unchanged throughout the duration of illness. That made no sense at all. Besides, the specific remedies had no data behind them for proof of efficacy, and were chosen on basis of charts and computer references.
One could hardly find anything about the paper that would lend credibility to its conclusion that suggested homeopathy “worked“ better than placebo. The results in our opinion demonstrated nothing more than the variations in the clinical trial system (noise.)
I recently learned of a study entitled “Dominican Children with HIV not Receiving Antiretrovirals: Massage Therapy Influences their Behavior and Development.” It disturbed me, and I couldn’t get it out of my head. They’re massaging these kids but letting them die of AIDS? I went back and read the complete article, and it left me even more disturbed.
They studied 48 Dominican children ages 2-8 with untreated HIV/AIDS, randomizing them to receive twice weekly sessions of either massage or play therapy for 12 weeks. The abstract said that those in the massage group improved in self-help abilities and communication, and that children over the age of 6 showed a decrease in depressive/anxious behaviors and negative thoughts. That’s what the abstract said. The text revealed a more complex story. (more…)
Homeopathy amuses me.
Well, actually it both amuses me and appalls me. The amusement comes from just how utterly ridiculous the concepts behind homeopathy are. Think about it. It is nothing but pure magical thinking. Indeed, at the very core of homeopathy is a concept that can only be considered to be magic. In homeopathy, the main principles are that “like heals like” and that dilution increases potency. Thus, in homeopathy, to cure an illness, you pick something that causes symptoms similar to those of that illness and then dilute it from 20C to 30C, where each “C” represents a 1:100 dilution. Given that such levels of dilution exceed Avagaddro’s number by many orders of magnitude, even if any sort of active medicine was used, there is no active ingredient left after a series of homeopathic dilutions. Indeed, this was known as far back as the mid-1800’s. Of course, this doesn’t stop homeopaths, who argue that water somehow retains the “essence” of whatever homeopathic remedy it has been in contact with, and that’s how homeopathy “works.” Add to that the mystical need to “succuss” (vigorously shake) the homeopathic remedy at each dilution (I’ve been told by homeopaths, with all seriousness, that if each dilution isn’t properly succussed then the homeopathic remedy will not attain its potency), and it’s magic all the way down, just as creationism has been described as “turtles all the way down.” Even more amusing are the contortions of science and logic that are used by otherwise intelligent people to make arguments for homeopathy. For example, just read some of Lionel Milgrom‘s inappropriate invocations of quantum theory at the macroscopic level for some of the most amazing woo you’ve ever seen, or Rustum Roy‘s claims for the “memory of water.” Indeed, if you want to find out just how scientifically bankrupt everything about homepathy is, my co-blogger Dr. Kimball Atwood started his tenure on Science-Based Medicine with a five part series on homeopathy.
At the same time, homeopathy appalls me. There are many reasons for this, not the least of which is how anyone claiming to have a rational or scientific viewpoint can fall so far as to twist science brutally to justify magic. Worse, homepaths and physicians sucked into belief into the sorcery that his homeopathy are driven by their belief to carry out unethical clinical trials in Third World countries, even on children. Meanwhile, time, resources, and precious cash are wasted chasing after pixie dust by our own government through the National Center for Complementary and Alternative Medicine (NCCAM). So while I laugh at the antics of homeopaths going on and on about the “memory of water” or quantum gyroscopic models” in order to justify homeopathy as anything more than an elaborate placebo, I’m crying a little inside as I watch.
The Lancet, meta-analysis, and homeopathy
If there’s one thing that homepaths hate–I mean really, really, really hate–it’s a meta-analysis of high quality homeopathy trials published by Professor Matthias Egger in the Department of Social and Preventative Medicine at the University of Berne in Switzerland, entitled Are the clinical effects of homoeopathy placebo effects? Comparative study of placebo-controlled trials of homoeopathy and allopathy.
Humans have the very odd ability to hold contradictory, even mutually exclusive, ideas in their brains at the same time. There are two basic processes at work to make this possible. The first is compartmentalization – the ideas are simply kept separate. They are trains on different tracks that never cross. We can switch from to the other, but they never crash into each other.
When contradictory ideas do come into conflict this causes what psychologists call “cognitive dissonance.” We then typically will relieve cognitive dissonance, which is an unpleasant state, through the second process – rationalization. We happily make up reasons why the two conflicting ideas actually don’t conflict at all. People are generally good at rationalization. It is a supreme intellectual irony that greater intelligence often leads to a greater ability to rationalize with both complexity and subtlety, and therefore a greater capacity to maintain contradictory beliefs.
In fact the demarcation between science and pseudoscience is often determined by the difference between sound scientific reasoning and sophisticated rationalization.
While cognitive dissonance refers to a process that takes place within a single mind, it is a good metaphor for the contradictory impulses of groups of people, like cultures or institutions. I could not help but to invoke this metaphor when reading two editorials published in the same day in the New York Times.
A few days ago, while gathering information for last week’s post about intravenous hydrogen peroxide, I noticed this:
ACAM Supports NIH Decision to Suspend TACT Trial
September 3, 2008, Laguna Hills, Calif. — The American College for Advancement in Medicine, ACAM today announced its support for the National Institute for Health’s (NIH) decision to suspend patient accrual of the Trial to Assess Chelation Therapy (TACT) Trial until allegations of impropriety can be proven false. ACAM believes that the TACT trial represents a important milestone in assessing the role of chelation therapy in modern healthcare and respects the decision of the NIH.
ACAM continue to work with Dr Tony Lamas to answer the unfounded allegations of impropriety.
“We believe that the Office of Human Research Protection (OHRP) will find that the allegations are of a political nature. To serve the best interests of participants enrolled in the TACT trial and all patients and their physicians who seek answers about chelation therapy, we call for a swift end to the moratorium and resumption of the trial,” said Jeanne Drisko, MD, President of ACAM.
I alerted a few others, including Stephen Barrett of Quackwatch, who queried the news room of the National Heart, Lung and Blood Institute (NHLBI: the joint sponsor, along with the NCCAM, of the trial) and got this reply:
The investigators and institutions performing the Trial to Assess Chelation Therapy (TACT), in conjunction with their Institutional Review Boards, have temporarily and voluntarily suspended enrollment of new participants in the study. NIH has not issued any announcement or press release about this action. To contact the Office for Human Research Protections’ (OHRP) press office, call Pat El-Hinnawy, (202) 253-0458.
In yet another round of science by press release, a particularly unimpressive acupuncture study is making the rounds of the major news outlets proclaiming that acupuncture works. I guess that is a sort-of answer to my title question – why are so many scientifically worthless acupuncture studies being done?
Let’s take a look at this particular study to see why it is so weak. All I have to go on is the press release, since the study is not published. It was presented at a scientific meeting – which is legitimate, I just don’t have access to it. (The bar for publication in a peer-reviewed journal is much higher than presentation at a meeting, and there may, in fact, be changes to the text prior to publication.) But we can still say a great deal about this study from the information provided.
Note: The reason that I am posting today rather than my usual Monday slot is because the article I discuss here was embargoed until last night. Consequently, I asked Harriet if she would trade days with me this week, and she was kind enough to do so.
One thing that science relies on almost absolutely is transparency. Because one of the most important aspects of science is the testing of new results by other investigators to see if they hold up, the diligent recording of scientific results is critical, but even more important is the publication of results. Indeed, the most important peer review is not the peer review that occurs before publication. After all, that peer review usually consists of an editor and anywhere from one to four peer reviewers on average. Most articles that I have published were reviewed by two or three reviewers. No, the most important peer review is what occurs after a scientist’s results are published. Then, all interested scientists in the field who read the article can look for any weakness in methodology, data analysis, or interpretations. They can also attempt to replicate it, usually as a prelude to trying to build on it.
Arguably nowhere is this transparency quite as critical as in the world of clinical trials. The reason is that medications are approved on the basis of these trials; physicians choose treatments; and different medications become accepted as the standard of care. Physicians rely on these trials, as do regulatory bodies. Moreover, there is also the issue of publication bias. It is known that “positive” trials, trials in which the study medication or treatment is found to be either efficacious compared to a placebo or more efficacious than the older drug or treatment it is to replace, are more likely to be published. That is why, more and more, steps are being taken to assure that all clinical trial results are made publicly available. For example, federal law requires that all federally-funded clinical trials be registered at ClinicalTrials.gov at their inception, and peer-reviewed journals will not publish the results of a clinical trial if it hasn’t been registered there. Also, beginning September 27, 2008, the US Food and Drug Administration Amendments Act of 2007 (FDAAA) will require that clinical trials results be made publicly available on the Internet through an expanded “registry and results data bank,” described thusly. Under FDAAA, enrollment and outcomes data from trials of drugs, biologics, and devices (excluding phase I trials) must appear in an open repository associated with the trial’s registration, generally within a year of the trial’s completion, whether or not these results have been published. Although there are some practical issues over this law, for example determining how much information can be disseminated this way without constituting prior publication, which is normally a reason to disqualify a manuscript from publication.
Yesterday was a good day.It was a good day because it was one of the days that shows that, sometimes, science and ethics do win out after all:
CHICAGO (AP) — A government agency has dropped plans for a study of a controversial treatment for autism that critics had called an unethical experiment on children.
The National Institute of Mental Health said in a statement Wednesday that the study of the treatment — called chelation — has been abandoned. The agency decided the money would be better used testing other potential therapies for autism and related disorders, the statement said.
The study had been on hold because of safety concerns after another study published last year linked a drug used in the treatment to lasting brain problems in rats.Chelation (kee-LAY’-shun) removes heavy metals from the body and is used to treat lead poisoning. Its use as an autism treatment is based on the fringe theory that mercury in vaccines triggers autism — a theory never proved and rejected by mainstream science. Mercury hasn’t been in childhood vaccines since 2001, except for certain flu shots.
But many parents of autistic children are believers in the treatment, and NIMH agreed to test it.The researchers had proposed recruiting 120 autistic children ages 4 to 10 and giving half a chelation drug and the other half a dummy pill. The 12-week test would measure before-and-after blood mercury levels and autism symptoms.The study outline said that failing to find a difference between the two groups would counteract “anecdotal reports and widespread belief” that chelation works.
Except that it wouldn’t have.