Sep 24 2009
CAM and Fibromyalgia
One of the common themes regarding alternative medicine is the reversal of normal scientific thinking. In science, we must generally accept that we will fail to validate many of our hypotheses. Each of these failures moves us closer to the truth. In alternative medicine, hypotheses function more as fixed beliefs, and there is no study that can invalidate them. No matter how many times a hypothesis fails, the worst that happens is a call for more research.
Sometimes this is the sinister and cynical intent of an alternative practitioner—refuse to let go of a belief or risk having to learn real medicine. Often, though, there are flaws in our way of thinking about data that interfere with our ability to understand them.
This week, the New York Times had a piece on alternative therapies for fibromyalgia. First a little background.
Fibromyalgia syndrome is a poorly-understood and controversial pain syndrome. In brief, it identifies patients who have significant chronic pain which is not due to any identifiable pathology. It probably includes a heterogeneous group of problems, but our understanding is limited. There may be changes in the way the nervous system deals with pain signals, but even this is not yet clear. It’s a disorder that can be very frustrating to treat, and even more frustrating to have. It is often co-morbid with depression, and the pain can be quite resistant to treatment.
Some practitioners deal with this by rejecting the diagnosis as being vague and useless. Others use the limited evidence we have to develop a treatment plan. And yet others turn to alternative medicine, and that is the topic of the Times piece. The article is a brief presentation by an expert with Q&A in the comment section.
One exciting area of research in the past decade has been in the realm of complementary and alternative medicine, or CAM, treatments for fibromyalgia. These range from well recognized therapies like acupuncture and massage to more novel treatments like d-ribose and qi-gong.
As this research grows, it is increasingly possible to identify CAM therapies that have some evidence of efficacy and minimal risk that can be incorporated right along with the more conventional treatment recommendations.
This is the typical claim of alternative medicine: it’s relatively harmless, and might even help. But what does the evidence say, and what are we to make of it?
One of our goals here at Science-Based Medicine is to recognize that in traditional evidence-based medicine it is easy to become overly reliant on the results of randomized-controlled trials (RCTs). While EBM does take into account the concept of plausibility, this is often lost when the data is “hot”. A paper looking at CAM therapy for fibromyalgia was recently released, and can serve as an example of how to think about these problems.
By way of introduction, let’s look at the abstract:
Best evidence was found for balneotherapy/ hydrotherapy in multiple studies. Positive results were also noted for homeopathy and mild infrared hyperthermia in 1 RCT in each field. Mindfulness meditation showed mostly positive results in two trials and acupuncture mixed results in multiple trials with a tendency toward positive results.Tendencies for improvement were furthermore noted in single trials of the Mesendieck system, connective tissue massage and to some degree for osteopathy and magnet therapy. No positive evidence could be identified for Qi Gong, biofeedback, and body awareness therapy.
This study looked at other studies to see how many showed promise in CAM therapies. The authors’ conclusions were based on the “positivity” of RCTs, that is, a modality was seen as possibly effective if the RCTs supported the hypothesis. This was despite the finding that overall, the studies were of mediocre quality. There are a number of flaws in this approach.
The authors state explicitly their reliance on the reputation of RCTs:
Most of the represented publications review both randomized controlled trials (RCTs) and non-RCTs. Though RCTs are considered as the strongest research basis for clinical recommendations in evidence-based medicine, RCTs are particularly difficult to perform in many categories of CAM medicine. An individualized approach to
the patient in diagnosis and therapy is often already part of the healing process itself. This makes standardization and the creation of control groups in order to rule out so called ‘placebo effects’ often very challenging, and blinding of both patients and medical practitioners sometimes impossible. Thus, it is not surprising that many authors focus on different study designs to fully cover the field. However, as RCTs are considered to be less liable to bias, there is also need for publications that focus on only these kinds of clinical trials. In 2002, a methodologically impressing publication appeared that covered RCT research on non-pharmacological approaches in fibromyalgia [24].
The authors re-iterate the special pleading used by CAM advocates to avoid being subject to scientific investigation, but decide to focus on RCTs to avoid the issue. What they fail to do is explicitly state what RCT results mean, beyond being positive and negative.
Plausibility
Dr. Harriet Hall would remind us of Tooth Fairy Science. We can measure all of the important data about the tooth fairy, including average get per tooth, average age of visitee, etc, but if we forget to question the fairy’s existence, we have failed to ask the most important question. It may be true that an RCT showed improvement in fibromyalgia patients using homeopathy, but since homeopathy is water, there is no reason to expect causality, and the results may be better explained by some other phenomenon.
This is explained mathematically by Bayes’ Theorem. If the prior probability of a positive result being due to the intervention is very low (say, because of implausibility), then any positive finding is very likely to be due to chance rather than causality.
Confounding natural variation with causation
Fibromyalgia is a syndrome whose symptoms naturally wax and wane. It can be very easy to confuse a change in disease state that occurs during a study with an actual effect. Rigorous controlling can minimize this but not prevent it. If, by chance alone, subjects in the treatment group had improvement in their disease due to its natural history, this will look statistically like a “win”. This makes the study of such disorders difficult, and opens a big door for CAM, as it is easy to convince others to follow your misattribution of cause. This is similar to concepts such as lead-time bias and regression toward the mean.
Built into this is the common cognitive error of confirmation bias. If you are a believer in the intervention, you may be prone to attribute positive results to the intervention even if there is no causation.
Damned statistics and replicability
The statistical tools we use to interpret RCTs are designed to help us tell systematic variations in the data from chance alone. There are a number of arbitrary assumptions built into this system. For example, if results are described by a normal distribution, we may define “abnormal” as the highest and lowest 2.5% of results. If a single RCT shows statistically promising results (say, >2.5 SDs from the mean), then it’s “positive”—but this still may be due to chance alone. A well-designed study can minimize the chance of this result being due to chance alone but cannot eliminate it. This is one of the reasons a single positive test for a less plausible hypothesis must be replicated before we get too excited.
The bottom line
Fibromyalgia is a complicated syndrome whose very nature makes it susceptible to the abuses of CAM practitioners. When evaluating a therapy for a complex disorder whose natural history is variable, we must very carefully parse out causation from correlation, recognize our own biases, and remember that a positive result of a randomized-controlled trial does not necessarily confirm a hypothesis. If an intervention has no plausible way or working, any positive results are likely a statistical artifact. Science is hard work, but the results are worth it.
___________________________________
References
Baranowsky, J., Klose, P., Musial, F., Haeuser, W., Dobos, G., & Langhorst, J. (2009). Qualitative systemic review of randomized controlled trials on complementary and alternative medicine treatments in fibromyalgia Rheumatology International DOI: 10.1007/s00296-009-0977-5
243 Responses to “CAM and Fibromyalgia”
Fibromyalgia is a very helpful diagnosis in that it immediately notifies the astute practitioner that the patient is crazy…
@manixter,
I can’t agree with that statement, however lightly it may have been intended.
That statement implies that there is nothing wrong with the patient other than a delusion; the patient likely has a legitimate problem that causes them real suffering, and even if that problem is pure delusion, the suffering (even if it is just psychological) is real, and the attitude is not helpful.
People cling to other similar diagnoses for idiopathic suffering like Morgellon’s, Chronic Lyme, and Gulf War Syndrome partly because they feel having a name for their condition gives them some power over their problem, and they feel that attempts to invalidate the diagnosis is an attempt to invalidate or deny their suffering and take that power away.
One thing that SMB skeptics/ critical thinkers need to always keep in mind is to not deny someone’s suffering while questioning the diagnosis/theory of the cause of that suffering, however hard that may be to do.
“An individualized approach to the patient in diagnosis and therapy is often already part of the healing process itself.”
I don’t think anybody that uses such a statement realizes it can be interpreted as an acknowledgment that nearly all CAM is an elaborate placebo.
” so called ‘placebo effects’ ”
What’s with the so called and single quotes?
You’ve got to hand it to these advocates of so called ‘alternative medicine’
“a positive result of a randomized-controlled trial does not necessarily confirm a hypothesis.”
No, it doesn’t, but repeated positive results might.
“If an intervention has no plausible way or working, any positive results are likely a statistical artifact.”
If you don’t understand how or why an intervention works, that doesn’t necessarily mean it doesn’t work. You can’t ignore RCTs and statistics merely because you don’t agree with them.
Wondering, you may need to read it more carefully, or have me explain it better, or read the links on prior probability.
Correlation does not prove causation. While there may be a causal link between two variables, certain things make it less likely, and some RCTs can, in fact, be ignored, or at least properly interpreted.
If the prior probability of something working (say, fairy dust for melanoma) is very low, but you still get a positive result on an RCT, it is mathematically very likely that you have a false positive. It is also likely that there are other, better explanations for your results than fairy dust helping melanoma.
[...] is the original: Science-Based Medicine » CAM and Fibromyalgia :a-big-door, actual, and-massage, chance-alone, decade-has, disease, exciting-area, [...]
“Correlation does not prove causation.”
The purpose of experimental controls is to demonstrate causation. If the experimental group gets the treatment and the control group gets a placebo, and there is a statistical difference between the groups, then the treatment probably caused the difference. That is the whole point of doing RTCs.
If you do only one experiment, you cannot draw any definite conclusion based on inferential statistics. Depending on the cutoff, there may be a 1 percent chance the treatment is not effective. That is why RCTs must be replicated by other researchers.
“If the prior probability of something working … is very low, but you still get a positive result on an RCT, it is mathematically very likely that you have a false positive.”
Actually no. Let’s say your p value (the probability of the results occurring by chance) is .001 — then no matter what the prior probability might happen to be (and different people will disagree about prior probabilities), a false positive is still unlikely.
Of course a false positive is always possible, but the p value tells you how likely it is. And the more times your RCT is replicated, the less likely it was a false positive.
Prior probability can be taken into consideration, and it actually always is — no one devotes time, effort and resources testing a theory they believe is absurdly implausible. And no one bothers testing a theory that is highly plausible and already accepted.
The purpose of research is to test theories that someone thinks might be true. If you apply for a research grant you would not say “My theory is almost certainly true and I want to spend the next year finding out if it really is.”
You also would not request funding to test a ridiculous theory that you feel is implausible.
So prior probability is very often in the mind of the researcher. If hundreds of carefully controlled experiments have already shown your theory to be false, then yes it has a low prior probability.
But for questions that have not already been answered by experiments, prior probability is largely subjective and not quantifiable.
In regard to the use of placebos, this came from the New York Times.
“Social support and beliefs affect a patient’s ability to rebound from illness, Dr. McDiarmid added, pointing out that over half of the people who respond to antidepressants do so because of the placebo effect.”
http://www.nytimes.com/2009/09/20/us/20shaman.html?_r=1&em
So it would appear that the use of placebos is alive and well in science based medicine these days. Anti-depressants as a placebo are not as safe as homeopathy.
you are far too kind and optimistic. NCCAM is only one example of an agency tasked to testing absurdly implausible claims.
“NCCAM is only one example of an agency tasked to testing absurdly implausible claims.”
From your point of view maybe, but obviously not from the point of view of the researchers it funds. Scientists are not always in agreement, and in fact they often disagree. That’s how it should be.
@Karl Withakay: thanks for your response regarding the subtleties associated with a fibromyalgia diagnosis.
regarding the your comment about the placebo effect, however – one of my neuroscience professors (extremely distinguished in his field) once described the placebo effect as being a category that modern medicine lumps some phenomena into that they do not yet know how to explain.
there are many things that are unknown (probably much more than is known). we are at a continually expanding frontier, and must remain humble.
i’m not convinced that all phenomena experienced by humans can be measured and described by the tools developed by science, and i see no reason to assume why they should. i have a deep level of respect and awe for that which scientific inquiry has shown us so far. but, with all due respect (and there is much due, of course) – assuming that the scientific method + materialistic worldview can and will provide a complete explanation for everything sounds just as dogmatic to me as pure, blind faith in a religion. I think it is often quite implicit, though – many people do not realize that this is what they are doing. Most of us have been brought up in a highly technophilic society, and it can be difficult to see the subtle memes that shape our perspective.
The way that people use the concept of ‘prior probability’ to discount some research is sometimes a consequence of this standpoint.
thank you for a well-written article.
Another example of the Scientific use of placebo, showing that they work, and cause fewer problems than antibiotics.
Antibiotic treatment of acute otitis media in very young children
The authors of this paper point out that guidelines recommend prescription of antibiotics in children with severe acute otitis media and in those under 2 years of age with bilateral acute otitis media or acute otorrhoea. For most other children with acute otitis media, initial observation is recommended. Such prescribing may shorten the course of the illness but may tend to over treatment. Their prospective trial involved 168 children aged 6 months to 2 years with acute otitis media in 53 general practices in the Netherlands. Half were treated with amoxicillin 40 mg/kg/day and the other half with placebo.
After 3.5 years they found that acute otitis recurred in 63% of the amoxicillin-treated group and 43% in the placebo group. Subsequent referral for secondary care was necessary in 30% of both groups. Their conclusion was that antibiotics are overused in such patients and should be used more judiciously.
BMJ 2009;338:b2525dol.10.1136/bmj.b2525
A word about P-values,
Yes it is true that a low people value means the odds are low that your data occurred purely by chance, but the T-test doesn’t know how you took your data. It is possible that there are variables which affected the outcome of the data that are merely artifacts of how the data was taken. A good example of this was a paper which came out a couple of years ago that showed the results of a well known experiment on the concept of cognitive dissonance may well be due solely to an artifact in the analysis of the probabilities involved in the experiment. Steve Novella blogged about this paper if you want to look for it.
oh man its late, obviously i meant p value and not people value
superdave,
A low p value doesn’t tell you that the experiment was well designed. A stupid experiment can have a low p value, but it’s still stupid and does not answer the question it claimed to ask. But if an experiment is sensibly designed and the treatment group differs from the control group, and the difference is probably not because of random variance, then you can’t discard the result simply because you feel it’s implausible.
This is simply false. Even with a p-value of 0.00001, a false positive can STILL be the most likely explanation if the prior probability is sufficiently low.
Also false. Most CAM studies simply ignore prior plausibility. Whereas good scientists often test absurdly implausible theories (one of my friends in grad school was looking for Lorentz violation, which is about as absurdly implausible as it gets – and yes, he and our advisor agreed that it was absurdly implausible, but it still needed to be checked), AND test highly plausible and already accepted theories (the majority of the field of high energy physics is devoted to attempting to disprove the Standard Model).
“Even with a p-value of 0.00001, a false positive can STILL be the most likely explanation if the prior probability is sufficiently low.”
Only if the prior probability is known to be low because the hypothesis has been sufficiently tested. If 100 experiments say theory A is false, and 1 experiment says it’s true, then a false positive is likely no matter how low the p value. But if the prior probability is assumed to be low because the theory is not generally accepted, but has not been carefully tested, then a false positive is unlikely.
You can only calculate the prior probability if an adequate amount of high quality research has already been done. Otherwise it’s subjective and not quantifiable.
The prior probability can also be known to be low because, if it were true, it would invalidate a great deal of what’s known in the field – a situation which results in most experiments in the field in question effectively serving as an indirect test of the hypothesis.
This is very much the case for most of the typical CAM modalities. Homeopathy and reiki are the most glaring examples (being completely contradictory to the entire body of biology, chemistry, AND physics), but chiropractic and acupuncture also fall into the class of “if they’re correct then most of what we believe we know about anatomy, physiology, and biology is wrong”.
“The prior probability can also be known to be low because, if it were true, it would invalidate a great deal of what’s known in the field”
You mean if it contradicts certain assumptions, which is not the same as invalidating known facts. It doesn’t seem at all fair to discount any experiment that seems to verify any CAM treatment, just because the theory violates your sense of how things ought to be. Not that I believe in the CAM theories. But when RCTs seem to verify them, you have to accept it as evidence. Otherwise, there is no point in anyone doing RCTs.
Will you only accept evidence that confirms your prior assumptions? That would be the opposite of the scientific method.
I’m referring to things such as the fact that reiki working would require a new type of physical interaction, having a strong effect in macroscopic situations. Such an interaction would be readily observed in other experiments (e.g. those measuring the parameters of the Standard Model); therefore the fact that it is NOT so observed is strong evidence that the prior probability is exceedingly low.
So no, I most definitely do NOT mean certain “assumptions.” I mean the fact that all sorts of experiments would already have detected qi/Innate/memory of water if in fact they existed and had the claimed effects. This constitutes exceedingly strong evidence that they do not, in fact, exist, and therefore the prior probability must be judged as infinitesimal.
Now, one might claim that there is some effect of (say) acupuncture that has only to do with poking the skin and not qi; however this is functionally equivalent to acknowledging that acupuncture is complete hokum – because the principles on which it is proposed to work, and the way it is applied, have no validity.
That might be relevant, if antidepressants were given as placebos. But they are not. A placebo, by definition, is biologically inactive. Antidepressants are given as active drugs because they have been found to work better than placebos.
To the extent that there is a placebo effect that is not due to statistical artifacts such as regression to the mean, the placebo effect may reinforce the biological effect of the active ingredient in a drug. That would certainly be a good thing, but it is not the reason for giving the drug.
By the way, what is the evidence for the “safety” of placebos? I find it curious that placebo advocates assign credit to the placebo for beneficial effects, yet hold it blameless for the long list of adverse effects, some of them serious, that routinely crop up in the placebo arm of controlled trials.
Assuming, of course, that there are no unrecognized biases or errors in the experiment. After all, a low p value only tells you that two treatment groups are likely different; it doesn’t tell you what that difference is. And assuming also and that the statistical assumptions upon which the calculation of p value is based are correct. For example, it is often implicitly assumed that he distribution of error is Gaussian, even when there is insufficient data to adequately test that assumption.
I’d speculate that it is a rare experiment when the likelihood of some kind of error is below 0.001. So one has to take very low p values with a grain of salt, particularly when the experiment has not been replicated, preferably by different investigators.
“one has to take very low p values with a grain of salt, particularly when the experiment has not been replicated, preferably by different investigators.”
I already said all that. A low p value simply means (if the statistics were done correctly) that the between group difference was probably not the result of random within group variance.
trrll:
I have to disagree with that. There are no ‘implicit’ assumptions. You either have Gaussian distribution or not. This needs to be tested every time before you can decide whether you’d use a parametric test or not.
wondering, what you say is reasonable:
However, as Scott says, plausibility determination prior to the study is very critical in later interpretation of the observations. A statistical significance (which is just a means of analyzing numbers) does not always predicate biological significance (which depends upon the interplay of physiological parameters). CAM theories like Reiki and Water memory are biologically implausible (based on our existing knowledge of anatomy, physiology, biochemistry and physics). Therefore, positive findings coming out of CAM studies must be subjected to close and greater scrutiny.
This is not unique to CAM; any time a new paradigm or hypothesis – that goes against existing knowledge – is propounded, in science it is always put up for stricter oversight, and parameters, such as repeatability of experiments, specificity of outcomes, confirmation from multiple approaches etc., are rigorously tested. Why should CAM be any different? If CAM therapies indeed work, why would they not be testable, why would they always need some sort of special pleading – mostly to justify their lack of difference in efficacy compared to placebo?
“Any time a new paradigm or hypothesis – that goes against existing knowledge – is propounded, in science it is always put up for stricter oversight”
Yes of course. I said that if 100 experiments showed that theory A is probably false, and one experiment showed it’s probably true, then theory A is still very much in doubt. But the original post said if a theory is implausible, then any experiment that seems to verify it is most likely a false positive.
That statement is much too vague and general. You have to define prior probabilities so they are quantifiable. How many experiments say “yes” vs. how many say “no?” What is the quality of the experiments?
That’s why meta-analyses are done, to gather the results of experiments done by many different researchers.
We can’t define prior probabilities as some kind of gut feeling that tells you “fairy dust can’t cure warts.” Science should consider the evidence, and gut feelings are suspect.
Wondering –
“That statement is much too vague and general. You have to define prior probabilities so they are quantifiable. How many experiments say “yes” vs. how many say “no?” What is the quality of the experiments?
That’s why meta-analyses are done, to gather the results of experiments done by many different researchers.
We can’t define prior probabilities as some kind of gut feeling that tells you “fairy dust can’t cure warts.” Science should consider the evidence, and gut feelings are suspect.”
PM> I am not sure what you class as a gut feeling and what you mean by “the evidence”. What about CAM propositions that are so inherently silly that no scientist has ever bothered to test them out?
In fact, often the failure of proponents to perform simple, obvious, NECESSARY studies before announcing core claims is one of the reasons for not taking claims seriously, especially when the pseudoscientists also cut corners by looking for effects upon subjective and other unstable outcomes in tricky, indirect tests such as clinical studies, knowing that this is an easier way of obtaining a few “positive” results.
I maintain that there are usually sound reasons for the kind of implausibility we are talking about and that it is certainly extreme enough to cancel out high P levels in certain types of study.
Here is a “scientific” observation that everyone has made in their kitchens. Dilution and succussion NEVER enhances the biological or physical properties of a solution. Everyone knows this for a fact. No reader will hurry off to do the relevant experiments with their coffee or pharmaceuticals.
Yet this kind of universal human experience can be overlooked when we permit wrong impressions to prevail as to what “science” is. In truth, we don’t know what it is is — we only know that certain modes of thought are helpful in getting closer to truth about the world.
@Peter:
Quite right. Actually, it should be added that wherever and whenever such experiments have been performed, it has always shown that dilution, indeed, reduces the biological properties; for example, when an antimicrobial drug is diluted serially, there is a point at which it stops working on its target microbe.
As far as I know, the homeopathic principle is supposed to work on the basis of molecular mimicry; the active principle or the drug is supposed to stimulate the same symptoms as the disease, and therefore, use of smaller quantities may actually sensitize or immune-modulate the host, so that the host mounts a stronger response to the disease condition.
What makes no sense absolutely is that weird principle of very large dilutions leaving virtually no molecule of the drug in the solution. A 6X dose reflects 10^6 times serial dilution, which probably is not so bad as far as the presence of the drug is concerned, but it is supposed to increase in ‘potency’ with further and further dilutions, which is a physical impossibility. Homeopaths often prescribe 30X and 200X doses!!
What also is extremely dubious is their insistence on a weird mind-body ‘axis’ for virtually every disease, including infectious ones, so that homeopathic medicines are ‘personalized’ for individual patients based on their ‘personality’ and ‘history’. In most cases, this represents a gross ignorance of physiology in health and disease, as well as pathogenesis.
By ‘Peter’, I meant Peter Moran – the commenter above mine.
“What about CAM propositions that are so inherently silly that no scientist has ever bothered to test them out?”
What about the germ theory of disease, which was considered so ridiculous and implausible that for a long time no one wanted to waste time testing it? Lots of things that we now accept seemed impossible and silly before they were tested. What could be more implausible than the theory of relativity, for example? Scientific progress is full of ideas that seemed ludicrous before they were tested.
Ah OzDigger another wonderfully cherry-picked and misrepresented nugget of propaganda. Did you actually read the literature to which the article so blithely refers? For a reasoned understanding and a more balanced view try here.
http://www.srmhp.org/0201/media-watch.html
But to summarise :
“Therefore, contra to some of the media “hype” on this topic, antidepressant research confirms an empirically demonstrated drug-placebo difference, although careful examination of this literature reveals that this difference is not nearly as large as most individuals believe, or as many of the pharmaceutical companies would have the public believe. Currently, the methodological problems with antidepressant trials preclude us from concluding definitively that the difference actually indicates specific biological effects of the drugs, as various nonspecific factors have not been adequately ruled out. Until these questions are answered, the media should understand that placebos can be double-edged swords, and that “expectancy” effects can result in harm as well as benefit. In a piece on this topic for the Guardian, a UK newspaper, Jerome Burne (2002) reports that many subjects in Leuchter’s trial (2002) relapsed and requested to be placed on the active medication after learning they were in the placebo arm. Vedantam’s Washington Post piece is similar to other articles on this topic that have appeared in the popular press recently, in that it occasionally betrays an imbalanced presentation of the evidence. The media should continue to follow this complicated debate and report on it responsibly, making certain not to overhype the “power” of placebo and, as a consequence, the “powerlessness” of antidepressants.”
It would appear that a proportion of people with diagnosed MILD depression show a response to depression that could be attributed to the placebo effect. For serious depression and other forms of psychosis the results show the medications are significantly more effective.
So what should this tell us? Well that we should not not rely on a single authority to provide proof of effectiveness. That some medications show promise in clinical trials and do not always fulfil that promise in post clinical studies ie the real world.
In other word that science based medicine is effective in proving what is clinically relevant and what is not. The fact that we perform rigorous and continuous challenges to established ideas and therapies allows us to continue to use what is effective and discard what is not. If something is shown to be not clinically useful ideas will change and new approaches will be developed.
That is the strength of the empirical method, its based on real world results. Sometimes we don’t like what those results tell us, but in the end as a community we have to accept reality and move on to something more effective or safer or wherever the evidence leads us.
And finallt to quote you
“Dr. McDiarmid added, pointing out that over half of the people who respond to antidepressants do so because of the placebo effect.”
As opposed to 100% of patients who take homeopathic preparations? When did the world of Homeopathy last challenge its preconceptions with real world evidence? When was the last time a therapy was abandoned or modified because of rational scrutiny of its effects?? Hell these things are not even tested for safety or efficacy BEFORE they are marketed, let alone followed up over the years.
The very fact that science based medicine continues to challenge itself is the best way forward we have, its not perfect and never will be. At least we can say we that have good reasons to believe something works, or to stop using it when its proved it doesn’t live up to expectations. What say homeopathy???
wondering,
“Scientific progress is full of ideas that seemed ludicrous before they were tested.” I”m wondering… would you recommend that we indiscriminately test every idea that comes along – first come first serve for research funds – or should we exercise some judgment in how to spend research dollars? And if you recommend using some judgment, what criteria would you suggest.
Wondering, I wanted to thank you for making an exceedingly good point eloquently.
Scott: “therefore the fact that it is NOT so observed is strong evidence that the prior probability is exceedingly low.”
The fact that X phenomenon has not been observed to date could also be because 1) no one has bothered looking, period, 2) no one has bothered looking with the right technology and instruments, 3) no one has bothered looking under the right conditions and in the right places, and/or 4) no one has bothered looking with the right operational definitions, research design, and methodology.
That is how science makes brand new observations never made before, which as you know, happens all the time.
It is only our personal investments in current paradigms that assume X is implausible or improbable because X does not fit well in said paradigms. But paradigms do shift. Anomalies considered implausible now may be explained by different premises and mechanisms we have yet to discover.
If one wishes to ignore anomalies until such time, go right ahead. But attacking good experimental research on anomalies just because they suggest our current paradigms are not perfect–well, that might be construed as narrow-mindedness.
“should we exercise some judgment in how to spend research dollars?”
It would obviously be ridiculous not to exercise judgement about what ideas and treatments are plausible enough to deserve funding. Theories that have been tested and convincingly discredited can usually be ruled out, unless the researcher provides good reasons for re-testing them in different ways. Theories that have not been well-tested but seem hare-brained are usually discarded.
But that’s when it becomes subjective. Different ideological groups and subcultures have different ideas about what is or is not plausible, and that’s where politics inevitably gets involved. Science can never live up to its ideal of seeking truth free of authoritarian constraints, because science needs money.
So we have CAM versus mainstream science, for example, fighting over resources. I don’t see how it could be otherwise.
> HelenSan: “But attacking good experimental research on anomalies just because they suggest our current paradigms are not perfect–well, that might be construed as narrow-mindedness.” wondering: “So we have CAM versus mainstream science, for example, fighting over resources.” <
You mean CAM versus science. There is no alternative science that CAM uses.
@ wondering on 26 Sep 2009 at 9:11 am “So we have CAM versus mainstream science, for example, fighting over resources.”
That only make sense if one subscribes to the idiotic (new-age, post-modern) notion that every idea is equally plausible. The NCCAM was established because sCAM notions cannot, factually, compete with scientific ideas.
It is no surprising that you are wondering, the world must be a confusing place when everything is mysterious and possible.
Gah, my reply to HelenSan got eaten. Trying again:
And post-modernism makes people 38% stupider but 74% more unaware of it. Perhaps we should research the anomaly of low intelligence and obliviousness among people who throw about terms like “current paradigms” or who suppose that “the right technology and instruments” will reveal water gains magical powers when you shake and dilute it a few thousand times.
The problem you are so blithely overlooking is this–even if something like homeopathy is plausible under “some other paradigm,” its proponents are claiming that they can tell it’s effective right here and now. You cannot say that a treatment can both be observed to have an effect on the body and that it cannot be tested for scientifically via controlled observation.
If research shows, without exception, that treatments like homeopathy, reiki and chiropractic, which work by implausible mechanisms, do not have real world effect beyond placebo (which they do not), then it’s not clinging to any “paradigm” to expect the pattern to extend to other similarly implausible treatments. Furthermore, if these “anomalies” you speak of are regularly amplified the more poorly controlled the research (which they are), that would indicate there is some effect of our psychology at work, and not some effect of the treatment.
“It is only our personal investments in current paradigms that assume X is implausible or improbable because X does not fit well in said paradigms.”
Occasionally this might be (approximately) true. Yet I suggest that among the sciences, medicine is is peculiarly prone to throw up what I have referred to as “inherently silly” alternative theories of illness. No other branch of human knowledge is so afflicted.
Have you never asked yourself why this should be so — and wondered about the likely reasons for this profusion of opposing theories?
Even if you haven’t, and wish to cling to the notion that some of the present crop might be true, you surely understand that they cannot ALL be true, except in the trivial but very explanatory sense that they all evoke placebo and other non-specific positive responses to medical interactions.
You surely also see why there are sound reasons why physicians should have a high level of skepticism about “alternative” ideas, especially those that conform to a usual stereotype as often outlined on this blog (a feature of which is too much reliance upon “it is only your biases that prevent you accepting our truths” rather than seriously trying to firm up the evidence).
The issue in my mind is not whether or not these methods can serve as medicine in many senses, — they clearly can. You are posing a different question with your appeals for more consideration /research i.e “do they have anything useful to contribute to medical knowledge?” The right answer is still “almost certainly not”, in some instances despite hundreds of years of trying.
[physicians should have a high level of skepticism about “alternative” ideas]
Yes they should, but they should also be skeptical about ideas that have been accepted by mainstream medicine. How skeptical were physicians when they prescribed HRT for women for decades, although it had not been thoroughly tested for safety or effectiveness? The reason medicine is prone to this kind of mistake is that patients are desperate for cures and treatments, and MDs would like to provide them. But in so many cases, nothing is available.
wondering on 27 Sep 2009 at 8:49 am “How skeptical were physicians when they prescribed HRT for women for decades …”
I guess they were skeptical enough to conduct a large study which suggested limiting the use of HRT.
“I guess they were skeptical enough to conduct a large study which suggested limiting the use of HRT.”
That was after prescribing it to millions of women for decades.
“That was after prescribing it to millions of women for decades.”
It had been tested for safety and effectiveness. It was prescribed because it worked for menopausal symptoms – better than any other treatment. Because it reduced the risk of osteoporosis. Because early studies indicated a possible benefit in cardiovascular disease. We knew there was a tradeoff between risks and benefits, and the recent large studies just shifted the balance further toward the risk side, leading to a change in practice. Incidentally, HRT does not increase the overall death rate: it increases the risk of some conditions and protects against others.
The lesson to be learned here is that science is self-correcting. Alternative remedies have not been given to millions of people for decades. They might turn out to have unexpected risks just like HRT. Alternative medicine does not do the kind of surveillance needed to pick up small risks. Alternative medicine has never said treatment X causes more harm than good so we’ll stop using it.
Scientific medicine could be described as organized skepticism. Alternative medicine is practically the opposite of that.
-pmoran: “physicians should have a high level of skepticism about “alternative” ideas”
-wondering: “Yes they should, but they should also be skeptical about ideas that have been accepted by mainstream medicine. How skeptical were physicians when they prescribed HRT for women for decades, although it had not been thoroughly tested for safety or effectiveness?”
What a stupefying bit of equivocation you’ve gotten up to here in your juxtaposition, wondering. Watching out for complications in established science-based medicine is nowhere close to being on the same level as maintaining skepticism towards wholly implausible ideas.
A. Noyd: “The problem you are so blithely overlooking is this–even if something like homeopathy is plausible under “some other paradigm,” its proponents are claiming that they can tell it’s effective right here and now. You cannot say that a treatment can both be observed to have an effect on the body and that it cannot be tested for scientifically via controlled observation.”
All right, let’s look at homeopathy as an example of CAM.
The “effectiveness” they claim uses the same operational definition that mainstream medicine uses: A close temporal correlation between the intervention and observed positive outcome. While mainstream medicine attributes this correlation to a widely accepted pharmacolgical mechanisms in the biochemical paradigm, homeopathy has no mechanism to offer as an explanation. That is the main difference: known mechanism vs unknown mechanism. The magnitude of the temporal correlations depends on study methodology and design.
When something is unknown, it is natural to offer other known mechanisms to explain the observations, such as the placebo effect. But what is wrong with the placebo effect? If one obtains the positive outcome desired for pennies on the conventional medicine dollar, why not use something cheap that can reliably produce the placebo?
And why preclude the possibility that future technology and more advanced instrumentation might reveal the heretofore unknown mechanism that would explain the “effectiveness” of homeopathy–either in addition to the placebo effect or instead of the placebo effect?
Weren’t tiny invisible creatures that go from person to person causing diseases considered ludicrous and implausible once upon a time–until microscopes were invented?
Wondering: “Science can never live up to its ideal of seeking truth free of authoritarian constraints, because science needs money.”
I think one step in the right direction would be to blind researchers to funding sources, and to make funding blindness the norm in science. (Michael Crichton proposed this for all politicized research, and I agree.)
A. Noyd: “Watching out for complications in established science-based medicine is nowhere close to being on the same level as maintaining skepticism towards wholly implausible ideas.”
I agree with Wondering. Skepticism needs to be objectively applied to both popular and unpopular ideas, results that we like with as well as results we don’t like. Isn’t that what science is about? Objectivity?
What is considered “science” is subjective. Most studies (short of RCT’s) published in medical journals sound like a load of drivel to me. You call it “science,” I call it pseudoscience that doesn’t meet any of the rigors and integrity of true scientific research, but goes through the motions and appearances. What’s that phrase Richard Feynman used? Cargo cult science?
So people who support medical cargo cult science calling CAM RCT’s “implausible” is, at best, like the pot calling the kettle black. At worst, it’s a disingenuous use of double standards.
The same, objectively defined standards for what science should be must be applied to all research. Whether one likes the ideology or implications or not.
Here is an example of medical research accepted by the mainstream that doesn’t meet my standards of good, true science.
http://freedom2question.blogspot.com/2009/05/what-is-pseudoscience.html
Good grief. Learn to read before you post.
Either you’re a complete idiot, or your entire post was a deliberate lie. Which one?
“Skepticism needs to be objectively applied to both popular and unpopular ideas, results that we like with as well as results we don’t like. Isn’t that what science is about? Objectivity?”
That is utter nonsense. Popularity has nothing to do with validity of an idea. Equal skepticism of results that we like and don’t like? We should be especially skeptical of results that we like.
HelenSan: “That is the main difference: known mechanism vs unknown mechanism. The magnitude of the temporal correlations depends on study methodology and design.”
No, the main difference is that science-based medicine can demonstrate via strictly controlled observation that its medicines and treatments are more than correlations and are effective beyond placebo. Homeopathy, like all CAM treatments with implausible mechanisms, cannot.
“When something is unknown, it is natural to offer other known mechanisms to explain the observations, such as the placebo effect.”
Do you even understand what the placebo effect is? If people getting fake homeopathy show the same level of relief as people getting “real” homeopathy, then the mechanism at work is indistinguishable from suggestion (or more likely is suggestion). Homeopathy thus is a placebo.
“But what is wrong with the placebo effect? If one obtains the positive outcome desired for pennies on the conventional medicine dollar, why not use something cheap that can reliably produce the placebo?”
You have a point only if the “desired outcome” from taking a medication or receiving treatment is always superficial relief of symptoms. There might be a place for placebos, but good luck finding a way to use them ethically since you have to lie to people to make them work. And if a condition calls for a medicine/treatment that does more than this, we’re SOL if we turn to things like homeopathy. Not so for SBM, despite its risks.
“And why preclude the possibility that future technology and more advanced instrumentation might reveal the heretofore unknown mechanism that would explain the “effectiveness” of homeopathy…”
WHAT effectiveness? Either it does more than placebo or it doesn’t! If it does, we can measure that in the here and now, regardless of our “paradigms.”
“Skepticism needs to be objectively applied to both popular and unpopular ideas, results that we like with as well as results we don’t like. Isn’t that what science is about? Objectivity?”
We’re not talking about popular vs. unpopular. We’re talking about medicine that is supported by plausible mechanisms and valid research vs. quackery that relies on implausible mechanisms and terrible research. My point to wondering is that skepticism over the safety of things that have a genuine more-than-placebo effect like HRT and skepticism towards things that have no plausible mechanism are two completely different things. Yes, both are necessary, but he’s implying they’re the same sort of thing. It’s like saying one must be equally skeptical about the reliability of vacuum cleaners and the existence of brownies who use magic to clean your house if you leave biscuits out for them.
“What is considered “science” is subjective.”
Only by people who have all their intelligence milked out by the stupidity of post-modern relativism. Science strives to minimize bias and elimiate circular reasoning. It works because it doesn’t merely support the things we want to believe in. You seem to have a warped understanding of science, either because you don’t understand it or you feel your beliefs are threatened by it, or both.
“So people who support medical cargo cult science calling CAM RCT’s “implausible” is, at best, like the pot calling the kettle black. At worst, it’s a disingenuous use of double standards.”
When CAM can coherently explain the mechanisms it supposedly operates by, accepts the results of unbiased tests of those claims, and shows its treatments have an effect beyond placebo, we’ll talk.
“The same, objectively defined standards for what science should be must be applied to all research.”
What objectively defined standards? Spit them out. I want to see what standards of science you use that excludes science-based medicine from being scientific.
wondering: as Scott and others have pointed out, the whole point of SBM is that it draws on knowledge gained from other disciplines. Physics is now complete as far as phenomena a human could experience (breakdowns in our knowledge occur at the centers of black holes, the beginning of the universe, and energy scales far, far hotter than the center of the Sun). There are some proposed treatments that contradict the known laws of physics. The point is, if one postulates, e.g., a “qi” force strong enough to interact noticeably with humans, this idea can immediately be dismissed without an RCT. Because we actually understand physics, chemistry, etc. better than that.
A. Noyd: “What objectively defined standards? Spit them out. I want to see what standards of science you use that excludes science-based medicine from being scientific.”
One principal feature of the scientific method is the use of well-controlled experiments. Medicine might even go through the motions of experimentation, but actually control for almost nothing at all. Sometimes having good controls is legitimately constrained by ethics. But other times, the only reason I can see for the absence of good controls is incompetence or duplicity.
Either way, if one either cannot have good controls for ethical reasons or choose not to have good controls for logistical or other reasons, the research cannot be called science. That is not to say the research doesn’t have value at all, but results from such research has to be very cautiously interpreted and even more cautiously generalized.
Ethics is not a valid excuse for lowering the standards so that uncontrolled crap can pass for “science.” Ethics just means that some problems cannot be scientifically studied, period.
For one example, I again refer to my previously linked article, “What is pseudoscience?”
http://freedom2question.blogspot.com/2009/05/what-is-pseudoscience.html
If you need other examples, give me the full text of any medical study you choose, and I will tell you what is wrong with it. But I’ll tell you how I do it, in case you care. I just read the study, write down the research design and structure, change the medical intervention to “homeopathy” or “pixie dust,” read it again, and voila–the flaws in the methodology and design come leaping out. It works like magic.
See, you guys KNOW what is wrong with these studies, because you rightfully apply them to CAM research. But what I have discovered is that most SBM proponents have a double standard. The very same flaws in conventional medical research are overlooked with a very forgiving eye. Skepticism is rigorously applied to results you guys don’t like, but leniently applied to results you do like.
All I am saying is skepticism should have no double standard, no preferential treatment for paradigm conformists.
Charon: “Physics is now complete as far as phenomena a human could experience … Because we actually understand physics, chemistry, etc. better than that.”
That is an assumption that some physicists I know would not agree with.
The assumption that we know almost everything there is to know, and that our paradigm will never shift is not a healthy one for the advancement of science. It will stagnate a methodology that is meant to be continuously self-correcting and dynamic.
Scott: [quote]Such an interaction would be readily observed in other experiments (e.g. those measuring the parameters of the Standard Model); therefore the fact that it is NOT so observed is strong evidence that the prior probability is exceedingly low.
Either you’re a complete idiot, or your entire post was a deliberate lie. Which one?[/quote]
You know, that is called a forced-choice fallacy.
I stand by my original statements. The assertion that “interactions” would be “readily observed in other experiments…measuring the parameters of the Standard Model” is only true if such interactions were looked for, looked for in the right time and place, and/or looked for with the right methodology and instrumentation. Etc.
Incidentally, it would make the debate environment more pleasant if I didn’t have to wade through ad hominem attacks (which are also fallacious). Thank you.
Harriet Hall:
“The lesson to be learned here is that science is self-correcting. Alternative remedies have not been given to millions of people for decades.”
correct. some forms of “alternative” medicine (i.e. Chinese medicine, Ayurveda, Tibetan medicine) have been used for millennia.
Harriet Hall: “Alternative medicine has never said treatment X causes more harm than good so we’ll stop using it.”
incorrect. there examples of substances in the Chinese pharmacopoeia whose use has been discontinued once they discovered that their potential toxicity outweighed their therapeutic benefits.
Harriet Hall: “Scientific medicine could be described as organized skepticism. Alternative medicine is practically the opposite of that.”
there’s a difference between skepticism and dogma. unfortunately the meanings of these two concepts are often transposed.
some forms of what you are calling alternative medicine and how it is practiced may be the opposite of organized skepticism, but you are generalizing. again, I draw your attention to the Chinese pharmacopoeia, a catalog of medicinal substances that has been consistently reviewed and revised for centuries, based on clinical efficacy and potential side effects. I realize that yours or another commenter’s definition of ‘clinical efficacy’ and how to measure this may currently be at odds with the metrics used by Chinese medical practitioners.
automato,
While millions may have used Chinese medicine, I question whether any single remedy has been used by “millions” for long periods the way HRT was.
I am intrigued by your mention of Chinese substances being discontinued from their pharmacopeoia because of potential toxicity. Can you specify which ones and provide references? I have been looking for examples of treatments that alternative medicine gave up on, but so far have not had any luck.
How were the remedies in Chinese medicine evaluated for clinical efficacy and potential side effects?
A. Noyd: “No, the main difference is that science-based medicine can demonstrate via strictly controlled observation that its medicines and treatments are more than correlations and are effective beyond placebo.”
RCTs can demonstrate effectiveness beyond placebo controls, true. But how exactly do they show that the effectiveness is more than a statistically significant temporal correlation between intervention and desired outcome?
” Homeopathy, like all CAM treatments with implausible mechanisms, cannot.”
But they have. A good number of homeopathy RCT’s have shown statistically significant temporal correlations (“effectiveness”) OVER the placebo control group. The problem is, those studies have flaws and the results are rightfully criticized.
My contention is that if SBM proponents use the same standards of criticism toward their own research, we might find less SB proof of effectiveness than previously believed.
“Do you even understand what the placebo effect is? If people getting fake homeopathy show the same level of relief as people getting “real” homeopathy, then the mechanism at work is indistinguishable from suggestion (or more likely is suggestion). Homeopathy thus is a placebo.”
Or homeopathy works as well as placebo. Elementary logic, my dear. If vanilla ice cream sales are the same as chocolate ice cream sales, it means vanilla and chocolate are equally popular flavors. It doesn’t mean vanilla and chocolate are the SAME flavor. See?
“You have a point only if the “desired outcome” from taking a medication or receiving treatment is always superficial relief of symptoms.”
A lot of conventional medicine works “superficially” as well. You have breast cancer, you cut out the lump. Three years later, you get it back, you cut out another lump. Same goes for headaches, and allergies, etc. If I can get that kind of superficiality without surgery or pharmacological risks, why not?
” There might be a place for placebos, but good luck finding a way to use them ethically since you have to lie to people to make them work. ”
You don’t have to lie. You can honestly say that this intervention has been observed to be temporally correlated with desired positive outcomes in thousands of people.
“And if a condition calls for a medicine/treatment that does more than this, we’re SOL if we turn to things like homeopathy. Not so for SBM, despite its risks.”
You can always try placebo and/or homeopathy BEFORE conventional medicine. If it doesn’t yield the desired outcome, most conditions have plenty of time to try more risky interventions. Hell, it takes that long to get a doctor’s appointment nowadays. While you’re waiting for your appt to come up, why not try something very cheap and possibly effective? If it doesn’t work, you can always go to your appointment.
“WHAT effectiveness? Either it does more than placebo or it doesn’t! If it does, we can measure that in the here and now, regardless of our “paradigms.””
Placebos are effective too, you know. If my migraine disappears because of the placebo effect, guess what? The migraine is still gone.
So desired outcomes do not necessarily have to work BETTER than a placebo.
Now the question of why there are not more RCTs demonstrating a higher correlation between intervention and positive outcome in homeopathy is a legitimate criticism. Homeopaths would say that the outcome is more dependent on the skill of the practitioner than conventional medicine. A good RCT design would use the same practioner to control for skill. But I have not seen that kind of study to date.
However, that is beside the point Wondering was making. IF a well controlled RCT with relatively few flaws were to demonstrate positive results for homeopathy, then those results should be acknowledged regardless of its “implausibility.” In science, one proceeds with replication using the same exact design and definitions before widespread acceptance. But to dismiss it from the get-go just because of the content, well, that betrays the objectivity of science.
“We’re not talking about popular vs. unpopular. We’re talking about medicine that is supported by plausible mechanisms and valid research vs. quackery that relies on implausible mechanisms and terrible research. ”
And what is considered “plausible” and “valid” is a function of its popularity.
“My point to wondering is that skepticism over the safety of things that have a genuine more-than-placebo effect like HRT and skepticism towards things that have no plausible mechanism are two completely different things. Yes, both are necessary, but he’s implying they’re the same sort of thing. ”
He’s implying skepticism in both questions requires the same standards.
“It’s like saying one must be equally skeptical about the reliability of vacuum cleaners and the existence of brownies who use magic to clean your house if you leave biscuits out for them.”
If you leave biscuits out and wake up to a clean house, then yes, the hypothesis that house-cleaning brownies can be beckoned with biscuits is worth testing. It doesn’t mean they exist, but just because one finds the idea ludicrous is not scientific proof that they don’t. Science will test it out methodically.
And let’s say, it turns out it’s not brownies but a friendly neighbor who smells the biscuits and sneaks in to eat them, then cleans your house out of guilt for B&E. Then there is the question, if it works, why not? Does it really matter, practically speaking, if you call the house-cleaner “brownie” or “Betty”?
“Science strives to minimize bias and elimiate circular reasoning.”
Well, that’s my point exactly. It eliminates bias by applying the same standards to everything, not just things we like. That is why I say skepticism cannot have a double standard.
“When CAM can coherently explain the mechanisms it supposedly operates by, accepts the results of unbiased tests of those claims, and shows its treatments have an effect beyond placebo, we’ll talk.”
1) Absence of a known mechanism does not invalidate effects. Those are two separate things. You can use a computer without knowing exactly how it works. Many psychotropic meds work without anyone understanding exactly what they do.
2) Agreed–so long as tests of those claims are unbiased and attempt to replicate the original finding using identical definitions, design, and methodology.
3) Again, what is wrong with as much effect as placebo? Let’s say Medication X relieves migraines for 6 months before they returns. Homeopathy remedy Y relieves migraines for 2 months before they return, and placebo relieves migraines for 2 months as well. OK, Medication X is more effective than either placebo and Remedy Y. But it also carries more risks. So why shouldn’t a consumer say, “You know what? I’m going to take Remedy Y every 2 months. Sure I have to take more of it than Med X, but it’s still cheaper in the long run, not to mention safer. What do I care as long as the migraines go away?”
I don’t want to sidetrack into defending CAMs. My point is that science needs to be objective about significant effects, whether we understand them or not, whether their mechanisms are known or not, whether they work psychologically or not. An effect is an effect, and science shouldn’t care from whence it comes.
“The very same flaws in conventional medical research are overlooked with a very forgiving eye.”
Again sometimes correct, but still shy of the point in relation to plausibility and its influence upon how much weight we apportion to different kinds of evidence.
There is usually no plausibility problem with pharmaceuticals and medical procedures, and to a lesser extent, with herbs — certainly nothing near that applying to medical systems that rely on undemonstrable, entirely speculative, processes or forces and that look like placebo in all other respects.
Our awareness of the innumerable possible flaws within the design, conduct, integrity, and interpretation of certain types of clinical studies makes it very difficult for highly implausible ideas to validate themselves SOLELY via that means.
Plausibility simply means taking into account ALL the relevant evidence.
HelenSan disagreed with Charon’s statement that “Physics is now complete as far as phenomena a human could experience.”
She said “That is an assumption that some physicists I know would not agree with.”
I had the same initial reaction when I read “physics is now complete” but then I realized the statement was qualified by limiting it to phenomena on a human level. I think that with that qualification, it is essentially true. We understand mechanics, optics, etc. fully enough to use the principles to predict accurately and use the knowledge effectively.
“The assumption that we know almost everything there is to know, and that our paradigm will never shift is not a healthy one for the advancement of science.”
No one is assuming any such thing.
So far the only effective traditional Chinese medicines that spring to mind, and I admit limited knowledge, are bear bile for gallstones and semen for peptic ulcers. You are welcome to use them of course. Herbals have been used in western medicine as well. I am old enough to remember learning Galenicals in medical school in Europe. A lot of our current medications have been derived from plants. The problem with herbals has been lack of quality control, contamination, inconsistent potency, due to season, growing conditions, etc. They did learn early on that you, for example, don’t drink hemlock extracts, or eat Amanita phalloides mushrooms. As far as I am concerned, our therapeutics, developed through scientific testing and knowledge of pharmacology, physiology, etc, are more consistently effective and superior to those that you mention and are the rightful heirs of the old standbys. Should we come upon hard times in the future, we may very well have to fall back on the herbals again.
“The assertion that “interactions” would be “readily observed in other experiments…measuring the parameters of the Standard Model” is only true if such interactions were looked for, looked for in the right time and place, and/or looked for with the right methodology and instrumentation. Etc.”
Now ask yourself “But, but — if these processes or forces have never been directly observed, why do we think they may exist? A valid scientific hypothesis has to explain something. ”
Of course, it all goes back to people claiming to be healed through the application of the undemonstrable technology.
Yet you have just now explained why proper controls are needed in RCTs, so you already understand what a shaky basis this is for a novel therapeutic hypothesis — unless, of course , consistent, unmistakable, replicable, objective effects of some kind can be demonstrated.
This is the way out of what you mistakenly regard as an impasse unjustly created by biased skeptics. Without this AM methods still look like differnt flavours of placebo medicine. Not that that is necessarily all bad.
HelenSan: “One principal feature of the scientific method is the use of well-controlled experiments.”
Can you explain “well-controlled” in such a way that doesn’t amount to “I’ll know it when I see it”? Because you have spelled out precisely nothing here, merely reiterated your opinion that SBM research is not rigorous. What I don’t know is if you have an understanding of real objective standards and, frankly, your nods to post-modern relativism make me doubt it very much.
“But how exactly do they show that the effectiveness is more than a statistically significant temporal correlation between intervention and desired outcome?”
Are you trying to ask if I think the beyond-a-placebo effects that show up in RTCs could be some sort of massive coincidence?
“A lot of conventional medicine works “superficially” as well.”
May I ask if you think any medicine or treatment has an actual effect beyond placebo (whether SBM or CAM)? If so, please give a few examples.
“You don’t have to lie. You can honestly say that this intervention has been observed to be temporally correlated with desired positive outcomes in thousands of people.”
If the patient believes that the “desired positive outcome” is something more than the placebo effect, then you are lying. And when you’re not outright lying, you’re denying your patient informed consent and playing on his/her misunderstanding of the difference between correlation and causation. That is ethically reprehensible.
“Absence of a known mechanism does not invalidate effects.”
My criteria were in response to this: “So people who support medical cargo cult science calling CAM RCT’s “implausible” is, at best, like the pot calling the kettle black.” The context was relative plausibility and now you’re talking effects. You seem to have a habit of shifting the context of our discussion in order to reply to my statements. For this reason, I’m ignoring most of what you said to me. Please don’t interpet this as my inability to answer your manifold responses, I merely feel it’s a waste of my time to counter responses that don’t actually address my point and it’s tiresome to correct you every time you shift the context.
No. That’s idiotic. Buningly stupid. It’s a conflagration of dumb. If something is no better than placebo, that doesn’t mean it’s “as good as” placebo, but that it’s as good as never having used it in the first place.
Harriet Hall: “I had the same initial reaction when I read “physics is now complete” but then I realized the statement was qualified by limiting it to phenomena on a human level. I think that with that qualification, it is essentially true. ”
And the physicists I know would disagree even with that qualification. Maybe especially with that qualification. I contend we know less about how physical phenomena interact with the human body than we know about how chemical phenomena interact with the human body.
“No one is assuming any such thing.”
Anytime someone says “physics is now complete,” I don’t see how this assumption (that we know almost everything there is to know, and that our paradigm will never shift) is not the underlying premise for such a statement.
Peter Lipson: “No. That’s idiotic. Buningly stupid. It’s a conflagration of dumb. If something is no better than placebo, that doesn’t mean it’s “as good as” placebo, but that it’s as good as never having used it in the first place.”
First of all, wow. Insult much?
“As good as never having used it in the first place” implies absence of intervention. But that is not at all true.
Why use a placebo control at all? Because researchers want to make sure the effects observed can be attributed to the pharmacological action and not a psychological one. Because psychological action is well documented in producing a positive effect mimicking pharmacological results. Because the placebo effect is an effect. It is NOT a baseline of no intervention at all.
Let me ask you this. What do you think you would find if RCTs had 2 control groups? One placebo control group and one do-nothing-at-all group? Would you expect a higher positive outcome in the placebo group than the do-nothing group? Of course you would. That is why the placebo effect is called a placebo *effect.*
Now if a CAM remedy can produce the same positive effect, however small, of a placebo, it is better than doing nothing at all. It would be as good as a placebo. Not as bad as doing nothing at all. See?
“Let me ask you this. What do you think you would find if RCTs had 2 control groups? One placebo control group and one do-nothing-at-all group? Would you expect a higher positive outcome in the placebo group than the do-nothing group? Of course you would. That is why the placebo effect is called a placebo *effect.”
I would not expect a better outcome in the placebo group for any objective measurements:
IS THE PLACEBO POWERLESS? An randomized of Clinical Trials Comparing Placebo with No Treatment N Engl J Med, Vol. 344, No. 21 • May 24, 2001
“We found little evidence in general that placebos had powerful clinical effects. Although placebos had no significant effects on objective or binary outcomes, they had possible small benefits in studies with continuous subjective outcomes and for the treatment of pain. Outside the setting of clinical trials, there is no justification for the use of placebos.
A. Noyd: “Can you explain “well-controlled” in such a way that doesn’t amount to “I’ll know it when I see it”?”
Sure thing. Well-controlled means you control for most, if not all, the major confounding variables, not just some of them. At least control for the confounding variables that would render the results completely meaningless.
My article at my blog, which I have linked to 2x, but which you obviously have not read, outlines some of these major confounders. Changing definitions of dependent or independent variables mid-study. Murky definitions. Excluding samples for murky and ill-defined reasons. Not getting a baseline–a real do-nothing baseline instead of treating the placebo group as a do-nothing baseline. Controlling with pharmacologically inactive substances rather than active ones. No statistical prestidigitation such as relative risk or adjusted values or person-years, which are easily massaged to yield desirable numbers. Or starting with an odds ratio design, but reporting a relative risk statistic, or vice versa. Etc.
Of course, major confounders in each study would be different, so it is hard to list confounders in general. If you really want to know what I mean about confounders, let’s take a study you think is good science, and I’ll rip it apart for you. Then you’ll see exactly what I mean by absence of rigor.
” I don’t know is if you have an understanding of real objective standards and, frankly, your nods to post-modern relativism make me doubt it very much.”
I apologize if anything I said led you to believe that I subscribe to any type of relativism. In fact, it is very much the opposite. I have very definite and specific ideas about scientific rigor that medical research doesn’t meet. When I say what is “science” in the mind of the beholder, I mean you look at the NEJM and see “science.” I look at the NEJM and see junk science. I am not saying your view is equally right as mine–which would be relativism. I am saying your identification of such publications as science is wrong, and my identification of such publications as crap is right–which is clearly not relativism. I hope I have made that point clear.
“Are you trying to ask if I think the beyond-a-placebo effects that show up in RTCs could be some sort of massive coincidence?”
Correlation does not mean causation, but it does not mean massive coincidence either.
I am saying proof of effectiveness is not equal to proof of causation. Two different things.
Unless the RCT shows an near 100% effect in the study group, you can’t infer causation. All you can infer is that the study group has a higher correlation than the placebo group. Causation is a complicated process that requires a series of mechanistic studies, and RCTs are not designed to address that complex.
“If the patient believes that the “desired positive outcome” is something more than the placebo effect, then you are lying. ”
Why do you have to make the patient believe that?
Patient has lifelong migraines. You say, “I know a bunch of people have tried X, and right after they have tried it, their migraines disappeared. It might not work for you, but it’s cheap, it’s as safe as water, and it’s worth a try, right?”
“And when you’re not outright lying, you’re denying your patient informed consent and playing on his/her misunderstanding of the difference between correlation and causation. That is ethically reprehensible.”
Why deny your patient informed consent? Tell him it’s untested if you want. Tell him there are plenty of very well tested alternatives if you want.
Here is the bottom line. Your patient with the migraine? He doesn’t care if it is correlation or causation. He just wants his migraines gone. And if it’s gone, everyone is happy. If it is not gone, he can try the next thing. What is ethically reprehensible about offering an alternative that is safe, cheap, and just might work?
Psychiatrists do it all the time with psychotropic meds, for example. They don’t know exactly how those meds work. The mechanism is not understood. But they sell them anyway, because their use is correlated with certain desired outcomes.
“The context was relative plausibility and now you’re talking effects. You seem to have a habit of shifting the context of our discussion in order to reply to my statements. ”
I am saying, what does known vs. unknown mechanism matter if the patient gets better (effect)? Why does “plausibility” matter if the patient gets cured? Because it certainly doesn’t matter to the patient.
As far as shifting context goes, it was not my intention. You bring up effects, I answer it. If you notice, I did wrap it all up in the plausibility context.
Mark Crisp: “I would not expect a better outcome in the placebo group for any objective measurements:”
Interesting article. Thank you.
So which is it? Is the placebo powerless? Or does it have the power to explain any positive effects of unknown mechanisms? If the placebo is powerless, than how would you explain it when CAM RCTs show positive effects over the control group?
“So which is it? Is the placebo powerless? Or does it have the power to explain any positive effects of unknown mechanisms? If the placebo is powerless, than how would you explain it when CAM RCTs show positive effects over the control group?”
Take 2 identical bottles of wine. Label one $5 and the other $100. The $100 dollar bottle tastes better and is enjoyed better.
Helenscan: “So which is it? Is the placebo powerless? Or does it have the power to explain any positive effects of unknown mechanisms?
If the placebo is powerless, than how would you explain it when CAM RCTs show positive effects over the control group?”
I think you are playing with us. You claim to be able to be able to find terminal flaws in any paper published in the NEJM, yet you wonder how we might explain the occasional positive results with usually subjective outcomes using implausible CAM methods? Are you implying a better standard of research in CAM?
WRT placebo, there is still much to learn. The evidence is consistent with a range of possibilities depending on the clinical setting. The Hrobjartsson et Al article that Mark referred you to cannot be taken as the last word, because the typical placebo-controlled RCTs upon which it is based seriously dampen placebo influences. For one thing the subjects are not expected to know if they are supposed to getting better or not. .
We do expect placebos to have very limited effects upon objective aspects of disease, as Mark says..
Utterly false. Such interactions would have direct effects on the parameters being measured, which would therefore not be as predicted.
Can’t have a debate if you refuse to read.
Calling idiocy “idiocy” is not an ad hominem fallacy but a statement of truth. It may not be a pleasant truth if you are the recipient, but it is still truth. Or, as one guy says, “A statement of fact cannot be insolent”.
Peter Lipson: “Calling idiocy “idiocy” is not an ad hominem fallacy but a statement of truth.”
Even if it is true, it furthers no argument of substance to the debate and adds to the decompensation of the debate environment. If this is the type of environment that you guys support, where dissent is subject to personal vituperation, I will happily bow out of here and leave you to your unchallenged rantfest.
pmoran: “I think you are playing with us. You claim to be able to be able to find terminal flaws in any paper published in the NEJM, yet you wonder how we might explain the occasional positive results with usually subjective outcomes using implausible CAM methods? Are you implying a better standard of research in CAM?”
Oh no! CAM research is just as bad if not worse. It is just that whenever a positive result is reported in CAM, the first thing I hear is “That’s just placebo.” So I am just wondering, if placebo is powerless, how did it have the power to produce all those positive results for which placebo was assigned credit?
Now if you hadn’t said “That’s just placebo,” and had explained those positive results with only methodological flaws to begin with, I wouldn’t be asking the question. It just seems inconsistent to me to attribute results to the power of placebo, then turn around and say the placebo is powerless.
“It just seems inconsistent to me to attribute results to the power of placebo, then turn around and say the placebo is powerless.”
Actually, some believe placebos are getting stronger, as in a recent Wired article and it was blogged here too.
“Our awareness of the innumerable possible flaws within the design, conduct, integrity, and interpretation of certain types of clinical studies makes it very difficult for highly implausible ideas to validate themselves SOLELY via that means.
Plausibility simply means taking into account ALL the relevant evidence.”
Thank you for acknowledging the possibility of innumerable flaws. I appreciate that–gives us a little bit of common ground.
Your definition of plausibility is more palatable to me. However, I still insist, to the patient, it is still irrelevant.
Example. True story with identity changed, obviously. Mr. X is having painful gallbladder attacks about 4 times a year for 1.5 years. The pain is excruciating and sometimes sends him to the ER for pain relief. The docs recommend gallbladder surgery but he can’t afford it. He tries homeopathy for 3 months, and lo and behold, his gallbladder attacks have stopped with no change in diet, going on 3 years and counting. Ultrasounds show he still has gallstones, but they don’t bother him anymore.
Guess what? Mr. X loves homeopathy. He doesn’t care that the mechanism is unknown and is likely to be the placebo effect. He doesn’t care that homeopathy is implausible in context of all the relevant evidence to date. He just knows he doesn’t suffer excruciating pain anymore, his trial with homeopathy was time limited, and he doesn’t even have to watch how much fat he eats.
If you ask him, he’ll just say, “Maybe they’ll figure out in the future how this works. Not knowing doesn’t take away from the fact that it worked for me. ”
Maybe that is why I keep harping about effects.
1) Why take away an alternative for the patient that can possibly relieve suffering–just because it doesn’t fit well into the current body of knowledge?
2) And to bring it back to Wondering’s point, why not acknowledge the positive effects when they do show up, objectively and impartially, as you would even if exactly the same thing happened with conventional medicine?
3) Why not allow that while it is currently implausible, it might not always be implausible as science advances in the future?
4) Why must “implausibility” trump the observation of effect, and automatically invalidate Mr. X’s relief and the entire CAM discipline?
Similar true story to Mr. X. Pain goes away without any treatment. Ultrasound reveals thickened GB wall but he repeatedly declines surgery. After several years of no symptoms develops jaundice and has a cholangiocarcinoma, inoperable, and dies from it.
HelenSan,
“It worked for me” is not an accurate statement. The patient may be committing the post hoc ergo propter hoc fallacy. He can only say “It was temporally associated with my improvement.” The only way to tell if a treatment really worked is to rule out both placebo response and improvement due to the natural course of disease by doing well-designed controlled trials.
@ HelenSan
Before I reply, I would request you answer this question, which you left out of your reply: May I ask if you think any medicine or treatment has an actual effect beyond placebo (whether SBM or CAM)? If so, please give a few examples.
Harriet Hall: ““It worked for me” is not an accurate statement.”
I agree. It isn’t. But what “work” means to scientists is different from what “work” means to patients. To them, “work” means temporal correlation with improvement. When they say the aspirin worked on my headache, don’t they simply mean to say, “The aspirin was temporally associated with improvement”?
Definition dispute aside, the fact remains Mr. X is pain-free for 3 years now. Even if that is a placebo effect, why deprive him from it? Why deprive a patient from the option of a cheap, relatively risk-free trial to be relieved of suffering?
Harriet Hall: “The only way to tell if a treatment really worked is to rule out both placebo response and improvement due to the natural course of disease by doing well-designed controlled trials.”
I agree completely. Really, thank you.
Unfortunately, such well designed controlled trials are rare even for conventional medicines. Show me conventional medicine study that controls for both the placebo effect AND the natural course of the disease–and I’ll show you 99 others that control only for the placebo effect (if at all), and very poorly at that. And yet those treatments are pronounced to have “worked.”
Helenscan: “Show me conventional medicine study that controls for both the placebo effect AND the natural course of the disease–and I’ll show you 99 others that control only for the placebo effect (if at all), and very poorly at that. ”
I don’t understand. EVERY double-blind placebo controlled trial controls for both placebo responses and the natural course of the illness. Such influences should be equal in each arm of the study and cancel each other out, leaving only any intrinsic effect of the treatment that is being tested.
What kind of error do you claim is prevalent within these studies?
If you really mean that such studies cannot tell us anything about the strength and frequency of placebo responses under the conditions of the study, then I am with you.
The study you refer to on your website is an epidemiologic study. The shortcomings are obvious when you compare it to a double blind placebo controlled trial. Unfortunately for your desires, parents are not too willing to engage their children in such trials and you are left with an imperfect study. I agree, that it is rare to find a jewel of a study in a medical journal but finding even a study of the type you criticize on your website in the CAM literature is like finding a palm tree growing in Antarctica.
Helenscan:
“Maybe that is why I keep harping about effects.
1) Why take away an alternative for the patient that can possibly relieve suffering–just because it doesn’t fit well into the current body of knowledge?
2) And to bring it back to Wondering’s point, why not acknowledge the positive effects when they do show up, objectively and impartially, as you would even if exactly the same thing happened with conventional medicine?
3) Why not allow that while it is currently implausible, it might not always be implausible as science advances in the future?
4) Why must “implausibility” trump the observation of effect, and automatically invalidate Mr. X’s relief and the entire CAM discipline”
PM> We are at the nub of it. The (presumed) best illustration you could think of for a CAM “effect” was a patient whose attacks of gallbladder pain stopped after many years and after taking a homeopathic remedy.
As a one-time gallbladder surgeon, I can offer several reasons why the attacks stopped when they did.
Gallstones are naturally erratic in their behaviour and this man might have another attack tomorrow.
The stones may have grown too big to travel and get impacted in the cystic duct or bile duct. The gallbladder may have become fibrosed and thickened from chronic inflammation and unable to contract. Rarely small stones may empty out, and he may not yet have grown a new crop — even this rare event is more likeley than that any one of several required processes of homeopathy are true. He is probably now very afraid of greasy foods and avoiding those may be contributing to fewer attacks.
Implausibility is not usually merely a matter of incompatibility with “existing paradigms”, there are nearly always other more likely explanations for the effects being claimed — often placebo responses, but in this case natural variation in the progress of the condition.
In Harriet’s recent Powerpoint presentation on the mechanics of science she included an interesting summary of the factors that need to be considered when making causal judgments. I can’t locate it on the blog at the moment, but she may post a link herself.
HelenSan said “Mr. X is pain-free for 3 years now. Even if that is a placebo effect, why deprive him from it? Why deprive a patient from the option of a cheap, relatively risk-free trial to be relieved of suffering?”
We have no way of knowing whether Mr. X was pain-free because of a placebo effect from the treatment or because he would have been pain-free anyway if he had had no treatment.
I have no objection to patients trying any treatment, however absurd. What I object to is providers lying to them, telling them something is effective when they have no good evidence that it is.
A. Noyd: “Before I reply, I would request you answer this question, which you left out of your reply: May I ask if you think any medicine or treatment has an actual effect beyond placebo (whether SBM or CAM)? If so, please give a few examples.”
I’m sorry I overlooked that.
There are plenty of drugs and CAM both that I “believe,” if you will, to have an effect beyond placebo. I have personally experienced some of these effects, from aspirin for a headache to slippery elm for a sore throat. I can only assume that millions of consumers of drugs like insulin or epidurals or CAM like herbs and acupuncture find them more effective than placebo as well.
Now have I seen scientific proof that any of these drugs or CAM are more effective than placebo? No. The inherent flaws are great enough that I can’t be certain of the positive result they report any more than you can be certain of the positive results you read in CAM journals.
First, they are often poorly written, so that the reader has insufficient information to judge the validity of the study. Maybe it really was a good study, but we can’t tell from the paper. For just one example, many papers do not operationally define their placebos. What exactly were those patients getting in their IVs or pills? Salt, sugar, tap water? Much of medical writing assumes the reader must trust the word of the researcher without any clear definitions. Some trials actually use pharmacologically active “placebos”!
Second, they often report a lot of “adjusted” statistics and almost never a straight summary of the raw findings. Again, the reader must trust that the researcher adjusted everything correctly, without any details on what adjustments were made or how to check up on those adjustments. Now this take-my-work-for-it vagueness appears to be standard in medical writings, but if you read papers in chemistry for example, they give you enough information (like what the raw findings were before adjustment) to independently verify their adjustments, if any.
Third, the comparisons between study and control group often lack controls for, or even discussion of, significant confounders. Sure they usually match up age, sex, race, and important disease variables. But most of the time, they overlook variables such as socioeconomic status, marital status, family support, stressors, health insurance, etc. Randomization is supposed to take care of variables one doesn’t think of, but these variables should have been thought of. Some of these variables are probably more significant than either sex or race for many diseases.
I can go on, but you get the point. Last, but not least, the study is rarely replicated. Sure, they do other studies on the same drug, but hardly ever using the same methodology or design. Replication of an exact methodology/design is an essential part of scientific methodology. But in medicine, one positive result often gets celebrity status and is hailed as “scientific” proof of effectiveness, and that’s it. Everyone then references that one study as cold, hard evidence. Real science, of course, is actually not so hasty.
So, even if I see a well-controlled RCT, I would want to see it replicated several times exactly before I feel it met criteria to be called “scientific” proof. You would expect the same out of CAM research.
I hope that answers your question.
PMoran: “I don’t understand. EVERY double-blind placebo controlled trial controls for both placebo responses and the natural course of the illness. Such influences should be equal in each arm of the study and cancel each other out, leaving only any intrinsic effect of the treatment that is being tested.”
Well, a placebo controlled study controls only for the placebo effect. It doesn’t control for what the course of the disease would look like had it not received any intervention at all. A good RCT should have a control group that comes in for the baseline measures and doesn’t come in again until it’s time for the end-point measures. THAT would tell you what the natural progression of the disease is supposed to look like.
I am saying you can’t assume that the placebo group is the same thing as a do-nothing group.
“What kind of error do you claim is prevalent within these studies?”
See my last response to A. Noyd.
“If you really mean that such studies cannot tell us anything about the strength and frequency of placebo responses under the conditions of the study, then I am with you.”
Yes, that too. Without a do-nothing baseline group, one has no idea what the placebo effect is, if it is even there.
PMoran and others,
Just to be clear, I was not offering the anecdote of Mr. X as absolute proof of causation or effectiveness. Of course, it could be complete coincidence with the natural progression of the disease, or the placebo effect, or whatever.
However, I don’t think the observation of a close temporal association between intervention and outcome should automatically be dismissed in favor of more mainstream explanations. It *could* be coincidence or placebo. But assuming that it absolutely *has to be* is another thing entirely. Jumping to the conclusion that the association cannot possibly be reflective of a true effect–well, that is not scientific.
In my view, science would not be hasty to dismiss and invalidate observations just because they don’t fit with the current paradigm. Science would be open to pursuing further research to see if those observations can be replicated, and then to systematically test hypotheses regarding those observations. How many Mr. X’s are out there? Let’s run a well-controlled RCT on this. Let’s replicate that RCT. Let’s hypothesize about possible mechanisms and test those hypotheses. That is how science works–not “Pffft. Another useless coincidence and superstitious patient to make fun of.”
And while science is painfully sorting it all out, I don’t see a problem with patients trying CAM out to see if something might “work” (be temporally associated with a good outcome) for them and relieve their suffering.
“Well, a placebo controlled study controls only for the placebo effect.”
Not in the least correct. The apparent “response” rate in the placebo control group of an RCT is the sum of a number of influences, at least these four —
1, True placebo responses (if these even exist under the conditions of the trial — in many clinical contexts they will be zero or negligible.).
2. Spontaneous improvements — would have happened anyway.
3. Biased reporting — subjects trying to give the nice doctors the right answer.
4. Other measures the patients may have adopted of their own accord.
If the study is designed and performed properly then these influences will cancel themselves out,
Helenscan: “– A good RCT should have a control group that comes in for the baseline measures and doesn’t come in again until it’s time for the end-point measures. THAT would tell you what the natural progression of the disease is supposed to look like.”
Not necessarily true. There will be a nocebo effect upon these people if they feel they are being denied treatment. Others may respond to the inevitable reassurance of being diagnosed, reassured and under medical observation.
It is more difficult than you think to get an entirely uninfluenced group of patients, when dealing with subjective complaints.
I think you are expecting too much of the most commmon kind of clinical study. It is designed to answer one single important question very precisely i.e. “does this treatment possess intrinsic efficacy?”.
If you want to answer other questions relating to the natural history of illness or placebo responsiveness or its effectiveness in average medical practice then certainly a different study design will be needed.
You seem to be criticising research for not answering questions it never attempted to answer.
.
Helenscan “And while science is painfully sorting it all out, I don’t see a problem with patients trying CAM out to see if something might “work” (be temporally associated with a good outcome) for them and relieve their suffering.”
Within limits I agree. But not about Mr X. I wish him luck, but he is 99.9% certain to still have gallstones and severe gallbladder pathology. He is at risk of serious illness at any time. If he has gone for three years without symptoms then I would be loathe to urge surgery upon him but he should know he is walking a tightrope.
I also cannot agree to extensive further research into homeopathy. It may have its place as simple safe placebo in societies that are accustomed to it. But it has had enough chances to show that it is anything more than placebo.
I don’t think you’ve adequately answered two of my questions. I’ll try to explain why, as well as briefly point out where you abuse relativism.
HelenSan: “I apologize if anything I said led you to believe that I subscribe to any type of relativism. In fact, it is very much the opposite.”
Getting rid of the babble about “different paradigms” would be a good start, then. And don’t flip between “plausible” and “popular” with the excuse that “And what is considered ‘plausible’ and ‘valid’ is a function of its popularity.” If we have both agree that we should use science to validate claims of effectiveness, then it ought to be understood we’re using science as a standard for plausibility. If you feel science cannot set a useful standard in this way and that the body of knowledge so far discovered can’t validly speak for the plausibility of things like homeopathy, then what are we using science for at all?
Here is more relativism and more context shifting in your reply to pmoran: “Your definition of plausibility is more palatable to me. However, I still insist, to the patient, it is still irrelevant.” Was pmoran arguing about plausibility in the minds of patients? No. The context was testing and overall plausibility with regards to “all the relevant evidence.”
“I am saying proof of effectiveness is not equal to proof of causation. Two different things.
Unless the RCT shows an near 100% effect in the study group, you can’t infer causation.”
This makes no sense at all. Proof of effectiveness requires causation, even if the mechanism is not what was expected or is not known. If X has effect Y, then it causes effect Y; it’s not just that X was correlated with effect Y. So when I talk about effectiveness, I am talking about causation. If an RCT failed to show causation then it also failed to prove effect.
“Correlation does not mean causation, but it does not mean massive coincidence either.”
Which doesn’t answer my question: Are you trying to ask if I think the beyond-a-placebo effects that show up in RTCs could be some sort of massive coincidence? If you’re not trying to ask this, what do you mean by “statistically significant temporal correlation between intervention and desired outcome” other than “coincidence”? Try giving a straight answer instead of being coy about correlation vs. causation. (This would be the first question.)
“I have personally experienced some of these effects, from aspirin for a headache to slippery elm for a sore throat.”
But you seem willing to dispute the causation on the part of aspirin. Do you or do you not think that aspirin had a causative effect beyond placebo on the relief of your headache?
“I can only assume that millions of consumers of drugs like insulin or epidurals or CAM like herbs and acupuncture find them more effective than placebo as well.”
I’m not asking about what you think seems to have an effect for other people in their own minds, I’m asking about what you think actually does have an effect (beyond placebo).
“I hope that answers your question.”
No, it really doesn’t. I want to know if you think any medications or treatments, whether CAM or SBM, have a causative effect (however redundant that seems to me, you apparently require it) beyond placebo. Not whether there are any that you “believe” have had an effect because that sort of answer leaves you weasel room under relativism where you could claim it wouldn’t have an effect for someone else operating under a “different paradigm” so it can’t truly be said to have an effect after all. Do you think there is any medicine or treatment that shows a causative effect beyond placebo where no one has jumped to conclusions? The proper answers are “yes,” “no,” or “I don’t know.” And then examples if the answer is yes. (This would be the second question.)
A. Noyd: Getting rid of the babble about “different paradigms” would be a good start, then.
I believe in an absolute truth. Reality exists independently of the human mind and objectively, yes. I also believe that we do not absolutely know that truth and reality right now. As we approach that absolute truth using the scientific method, we go through different paradigms. That is a fact in the history of science. It is not relativism. Medicine used to operate under religious and philosophical ones. Currently, it embraces the biochemical paradigm. Who knows what paradigm will come next that will reveal our current one to be inaccurate?
So what you THINK is plausible now, may be completely wrong. Plausibility is an opinion. Human knowledge is too incomplete to decide such opinions can be free from error.
“If X has effect Y, then it causes effect Y; it’s not just that X was correlated with effect Y. ”
I defined “effectiveness” differently from the outset as a correlation. Nothing more. The RCTs are so poorly controlled they cannot demonstrate causation, only correlation. So I interpret all medical RCT results as having only the confidence and validity of a correlation. You think it shows causation because you believe the research design is valid. I don’t.
“Try giving a straight answer instead of being coy about correlation vs. causation. ”
You say coy. I say precise.
“But you seem willing to dispute the causation on the part of aspirin. Do you or do you not think that aspirin had a causative effect beyond placebo on the relief of your headache?”
I think it. I have seen no scientific proof of it. I think many things for which I have no scientific proof, you see. I think this one particular restaurant gave me food poisoning. But I will never have good, hard proof of it. I think reading to my kids early on gave them a love of books. But do I have scientific proof of that? No.
“I want to know if you think any medications or treatments, whether CAM or SBM, have a causative effect (however redundant that seems to me, you apparently require it) beyond placebo. ”
Whatever I think, I have seen no scientific proof of a causative effect beyond placebo in the extensive medical literature I have sampled.
“Do you think there is any medicine or treatment that shows a causative effect beyond placebo where no one has jumped to conclusions? The proper answers are “yes,” “no,” or “I don’t know.””
NO. Absolutely not. None of the RCTs I have encountered in my readings have sufficient scientific rigor to demonstrate causation.
Pmoran: “If the study is designed and performed properly then these influences will cancel themselves out.”
I think you have identified one of our most fundamental differences on what constitutes scientific rigor. I see no methodology for these influences to “cancel” anything out. In scientific method, these influences are called confounders, and it is the researcher’s responsibility to design controls for ALL confounders. Confounders don’t just magically cancel themselves out.
It is unique to medical research to rely on randomization, large sample sizes, and whatever magic mechanism you have identified to “cancel” out confounders–rather than tediously design controls for each and every one of them. That is what scientists in other disciplines do. They don’t have magic confounder cancellation elves.
“There will be a nocebo effect upon these people if they feel they are being denied treatment.”
Then control for a nocebo effect. But regardless, a no-intervention control would reflect natural progression of a disease more than a placebo group would.
“It is more difficult than you think to get an entirely uninfluenced group of patients, when dealing with subjective complaints.”
My point is precisely that it is more difficult than you think. Which is why you need more than one control group. The confounders are too numerous to name, so at least control for the ones you can think of. Just because it is so difficult doesn’t mean it is ok to ignore all confounders but one.
“I think you are expecting too much of the most commmon kind of clinical study. It is designed to answer one single important question very precisely i.e. “does this treatment possess intrinsic efficacy?”.”
And I am saying without controlling for all the confounders that influence the measured outcome, we have no idea what the intrinsic efficacy is. For all we know, the differences between the study group and the placebo group could be because the placebo group was poorer, had less family support, and didn’t exercise. Now you can claim they weren’t poorer all you want, but you have no PROOF of it because it wasn’t even addressed or measured! A poorly controlled RCT relies solely on wishful thinking that both groups are exactly the same in every way that could influence the efficacy results.
“If you want to answer other questions relating to the natural history of illness or placebo responsiveness or its effectiveness in average medical practice then certainly a different study design will be needed.”
Then we agree at last. A different study design is needed! Thank you!!!!
What was the first thing you guys all pointed out when I gave the example of Mr. X? How do we know it wasn’t placebo or a natural progression? You can’t make those questions essential for determining efficacy of CAM, but optional for determining efficacy of a study drug. Well, you can, but science wouldn’t approve of the double standard and preferential treatment and absence of objectivity.
“You seem to be criticising research for not answering questions it never attempted to answer. ”
My criticism is precisely that they never attempted to answer those questions–especially when they demand those same answers from CAM.
Answering these questions (and more) is essential to show that the efficacy can be attributed to the study drug, rather than to placebo or natural progression. Without those answers, without the proper controls for all the major and obvious confounders, you have no proof of efficacy.
Medicine practices what I call “cookbook science.” You follow the recipe, put in the ingredients of randomization and placebo, pop it in the oven to bake, and out comes “Scientific Proof of Efficacy!” Medical schools do medical researchers a great disservice by not teaching proper scientific method, which is not a recipe but an understanding applied individually and carefully to every research question:
“There is no science without control.”
And the corollary…
“The less control you have, the less science you have.”
So let me see what you are saying. My simple mind has trouble dealing with abstractions. If you take 1000 Mr Xs that meet the criteria for surgery and in half you remove the gallbladder and to the other half you give your favorite herb or homeopathic potion and wait, let’s say 10 years, and see how many die in each group of gallstone related complications or have hospitalizations for such. If you had statistically significant differences between these groups, that would not be scientific proof because some of the patients that didn’t have surgery initially, had emergency surgery later on?
“How do we know it wasn’t placebo or a natural progression? You can’t make those questions essential for determining efficacy of CAM, but optional for determining efficacy of a study drug.”
Can you give an example of a study drug for which these questions are not essential?
Confounders tend to ruin studies showing perhaps a lower efficacy of a study drug. An example would be the FIELD trial. It was hoped to show efficacy of fenofibrate, but it couldn’t control for the increased use of statins, that became the standard of care during the study period. The results were a dud, making the trial worthless in my opinion.
Helenscan. I know what a confounder is, and it is something that applies when trying to derive causal associations from observational studies and anecdotal data.
The whole raison d’etre, the fundamental logic of of the modern placebo-controlled RCT is to “control for” known, and even unknown, confounders by ensuring that they will have equal influence in both groups, if the study is large enough and the subjects are properly randomised. This is what I meant by them “cancelling out”.
Do you not yet understand that?
Pmoran: “The whole raison d’etre, the fundamental logic of of the modern placebo-controlled RCT is to “control for” known, and even unknown, confounders by ensuring that they will have equal influence in both groups, if the study is large enough and the subjects are properly randomised. This is what I meant by them “cancelling out”.”
Medicine practices a false reliance of large sample sizes and randomization as inadequate substitutes for true controls. It’s lazy pseudoscience. And no, they do not allow confounders to “cancel out.” That is a lie taught only in medical schools and in no other scientific discipline.
“Can you give an example of a study drug for which these questions are not essential?”
Any study that has no do-nothing control group cannot answer the question about natural progression of disease.
“If you had statistically significant differences between these groups, that would not be scientific proof because some of the patients that didn’t have surgery initially, had emergency surgery later on?”
Maybe the group that got CAM had to pay out of pocket because their health insurance wouldn’t pick up CAM. Their finances suffered and their spouses left them. They got depressed and started drinking. They lost their jobs, and all that stress caused decompensations, which resulted in hospitalizations and deaths for gall bladder disorders.
A lot of things can happen in 10 years that influence the outcome. Without controlling for them in the study design, the study cannot scientifically attribute the outcome to the independent variable. A large sample size and randomization can *help* control for selection bias at the beginning, but they cannot control for many other confounders that influence the outcome during the course of the study.
“Medicine practices a false reliance of large sample sizes and randomization as inadequate substitutes for true controls. It’s lazy pseudoscience. And no, they do not allow confounders to “cancel out.” That is a lie taught only in medical schools and in no other scientific discipline”
What nonsense!
Do you really think laboratory scientists know every possible influence they are controlling for when they use a reagent blank in their test -tube experiments? This is the study design that clinical trials try to mimic.
Do lab scientists measure the day’s temperature, and reagent strength, and instrument sensitivities and try to “control” in some bizarre way for each one individually in every test run?
No. You have a lot to learn but are not listening.
pmoran on 02 Oct 2009 at 1:49 am “You have a lot to learn but are not listening.”
Yes, perhaps HelenSan should get a dictionary and look-up “sophomoric.”
Or, as someone I know likes to say “If you can’t understand; maybe it’s you: http://www.apa.org/journals/features/psp7761121.pdf ”
The article is titled Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments
Pmoran: “Do lab scientists measure the day’s temperature, and reagent strength, and instrument sensitivities and try to “control” in some bizarre way for each one individually in every test run?”
If the day’s temperature, reagent strength, and instrument sensitivity can confound (influence) the dependent variable (the measured outcome), YES–they design controls for those confounders as best as they can. If they cannot do it all in one experiment, they do it in successive experiments. Then in the later papers, they cite the controls found in previous experiments.
What they do not do is say “I will let my magic confounder cancellation elves, Big Sample and Random, take care of the confounders. They do such a good job I don’t even have to measure the confounders and make sure they are really gone!”
“No. You have a lot to learn but are not listening.”
Well, of course, I could say the same of you.
I am married to a PhD physical chemist who works as a research scientist for a national laboratory. We talk about scientific methodology and design all the time. When I read some of the responses on this thread to him, he rolled his eyes and said, “Why do you talk to these people? They don’t know what science is, and they are not going to start learning from you.”
I suppose we just have to agree to disagree about what constitutes science.
I’ll leave you with this article, the speech where Richard Feynman introduced the term, “cargo cult science.”
http://calteches.library.caltech.edu/51/2/CargoCult.pdf
Here are some noteworthy quotations:
1) “Nothing happened. So I was unable to investigate that phenomenon.”
Notice that when Uri Geller was unable to bend his spoon, Feynman doesn’t jump to the conclusion, ‘Aha! It’s all fake!” He concludes more precisely that he was unable to investigate it. He understands something you guys often forget, that absence of evidence does NOT equal evidence of absence.
2) “For example, if you’re doing an experiment, you should report everything that you think might make it invalid–not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you’ve eliminated by some other experiment, and how they worked–to make sure the other fellow can tell they have been eliminated.”
He is, of course, talking about what we call confounders, “other causes that could possibly explain your results.” You need to report them, report the controls you are relying on in previous experiments, etc–”to make sure the other fellow can tell they have been eliminated.”
Well, I’m the other fellow. When I read an RCT, I see no evidence that the confounders have been eliminated. Therefore I see no evidence to conclude efficacy or any interpretation that goes beyond the confidence of a correlational study.
You see, if I ask the question, “What if some placebo patients started using recreational drugs in the middle of the study, and those drugs were what caused a poorer outcome?”
You might say, “Well, I am sure some study patients started using recreational drugs as well. Because large samples and randomization makes sure both groups are equally likely to start using drugs in the middle of the trial.”
I say, “Well, that would be convenient if it were true. Do you have evidence that recreational drugs were used or not used in equal numbers in both groups? Did you at least ask them at the end of the study, so you can say, ‘both groups reported the same level of recreational drug use.’?”
You might say, “Oh no, I don’t need to ask. Everyone who is not ignorant just KNOWS large sample sizes and randomization eliminates the need for controls for every little possible confounder and evidence of controls.”
After reading the article, what do you think Dr. Feynman would say to that?
I’d be interested to know which comments in particular. Either you were cherry-picking, or he’s incompetent.
Scott: “I’d be interested to know which comments in particular. Either you were cherry-picking, or he’s incompetent.”
I am sorry I don’t follow. What comments?
So what you are saying is that medicine is a pseudoscience as no experiment or study can meet the criteria that you are imposing on it. OK. So your husband is a scientist and he controls all the confounders. Has he succeeded then in making 2 identical snowflakes? If he can’t do that, then according to your logic, he is practicing pseudoscience.
The ones you read to your husband.
@ HelenSan on 02 Oct 2009 at 10:22 am- Can’t you just make a point? I read all that just to discover that you don’t understand the function of large-scale and randomization. Then you propose the “straw man” that medical researchers do not look for confounding errors; they do.
HelenSan on 02 Oct 2009 at 10:22 am “I am married to a PhD physical chemist … When I read some of the responses on this thread to him, he rolled his eyes and said, “Why do you talk to these people? They don’t know what science is, and they are not going to start learning from you.””
My former colleague, now retired, used to teach a course we called “chemistry for poets” since it was meant to satisfy a science requirement for non-science majors. You and your husband would have benefited from it. He explained the spectrum of sciences from rigorously structured and easily controlled (physics) to highly complex and difficult to control (sociology). You two do not appreciate the complexity of research in medicine.
BTW, there are many accomplished scientists operating this blog and many of the MDs without advanced degrees in science, per se, understand it better than you and the hubby.
Scott: “The ones you read to your husband.”
I read all the comments posted (except my own) in their entirety, from 29 September onward.
Joe: “He explained the spectrum of sciences from rigorously structured and easily controlled (physics) to highly complex and difficult to control (sociology).”
The definition of science is very simple: a discipline that obtains knowledge by the use of the scientific method, which is: observation, hypothesis, experimentation, and theory. Experimentation is defined as a research design that measures how much influence one variable (independent variable) has on another variable (dependent variable), by controlling all other variables that could influence the dependent variable (called confounders). Poor control = poor experiment = poor science.
So if A, B, C, D, E, and X all have influence on dependent variable Y, and you want to isolate the influence of X on Y, you have to control for A, B, C, D, and E. Only after that can you reasonably conclude the results of Y could have come from only X. If you control for only A and assume that B, C, D, and E will just “cancel out” with no evidence that they did, you cannot conclude that B, C, D, and E did not cause all or part of the result you measured in Y.
Highly complex disciplines that cannot do experimentation and therefore control confounding variables rely on only one step of the scientific method: observation. They observe anecdotes and correlations, and do the best they can with that information. If they carefully qualify their interpretations in context of having no controls and incomplete application of the scientific method, I don’t have a problem with calling disciplines like sociology a quasi-science rather than a pseudo-science.
The accusation of pseudo-science is made when a quasi-science routinely tries to pass off poorly controlled studies as complete and reasonable proofs of causation.
And you are saying that medical studies are poorly controlled and incomplete and we are trying to pass them off as complete proofs of causation? Do you understand p values?
@HelenSan on 02 Oct 2009 at 1:08 pm “The definition of science is very simple: …”
You are more than 40 years late to lecture me on the basics of science. That’s why I suggested you look up the meaning of sophomoric, have you done that? Put another way, you know just-enough to be dangerous.
Weing: “So your husband is a scientist and he controls all the confounders. Has he succeeded then in making 2 identical snowflakes? If he can’t do that, then according to your logic, he is practicing pseudoscience.”
Controlling for confounders is an unending process. There will always be confounders you haven’t yet thought of or have limitations in controlling because of ethics, funding, limited technology, or other constraint. This is the difficult and complex nature of science, which is why scientific progress is painfully slow.
Pseudoscience occurs when researchers jump to conclusions that are unsupported by the quality of the controls. If the researcher makes no such leap and carefully qualifies the interpretation of the current study to match the data, he is not engaging in pseudoscience.
So for example, Dr. Smith runs an RCT on study drug X on how often MS patients relapse within 2 years. He controls for current medications, diagnosis, prognosis, age, gender, race, and placebo. But he does not even ask about diet, physical exercise, social and emotional support, significant life stressors (job change, divorce, relocation, death in family), use of recreational drugs, and use of CAM.
Imagine that Dr. Smith says, “My study showed a significant correlation between use of study drug X and a low relapse rate, controlling for medications, diagnosis, prognosis, age, gender, race, and placebo. But the low relapse rate amongst drug recipients could have been caused in part by group differences in diet, exercise, support, stress, substance abuse, and CAM, which I did not measure or address. Future studies should investigate the study drug while controlling for these confounders before making conclusions about efficacy.”
I would have absolutely no problems with Dr. Smith’s conclusions. They are consistent with the research design and the data obtained.
Now imagine that Dr. Smith says, “My study showed significant reduction in relapses when compared to a placebo group. The drug is effective in reducing the risk of relapse.”
I would say his conclusion is inconsistent with his research design and claims of effectiveness without evidence is tantamount to the practice of pseudoscience.
See, pseudoscience is revealed in the conclusions, not necessarily in the methods.
weing on 02 Oct 2009 at 1:20 pm “… Do you understand p values?”
I think you can trust her to know the technical term- “urinalysis”.
“Now imagine that Dr. Smith says, “My study showed significant reduction in relapses when compared to a placebo group. The drug is effective in reducing the risk of relapse.””
I think Dr. Smith can only say “My study showed significant reduction in relapses when compared to a placebo group.” The second statement is not warranted.
HelenSan: “As we approach that absolute truth using the scientific method, we go through different paradigms. That is a fact in the history of science. It is not relativism.”
As a pure fact, this works. However, you’re actively exploiting “different paradigms” to avoid confronting certain points, as though there’s no greater or lesser probability in what one set of people says vs. what another says being true. This doesn’t follow and this is where relativism sneaks in. You’re free to weasel out of straight answers and valid conclusions by putting forth the-argument-according-to-people-using-paradigm-X or reframing the question in the context of what-people-using-paradigm-X-believe and then refusing to entertain criticism or counterargument, saying without absolute knowledge, we can’t know which paradigm is more accurate. However, you believe we’re approaching absolute truth at the same time you insist we don’t know which paradigm is getting us there. This means your belief is a sort of unreasoned faith in progress.
“Medicine used to operate under religious and philosophical ones. Currently, it embraces the biochemical paradigm. Who knows what paradigm will come next that will reveal our current one to be inaccurate?”
The religious and philosophical “paradigm” wasn’t inaccurate, it was wholly wrong. The biochemical “paradigm” might be inaccurate in the sense that it’s incomplete and imprecise, but there’s no reason to think that it will be wholly supplanted by some other “paradigm.” It quite clearly works in a way previous religious or purely philosophical notions failed. If you wish to argue that it only seems to work because I’m invested in the biochemical “paradigm,” then that would be relativism.
“I defined ‘effectiveness’ differently from the outset as a correlation. …You think it shows causation because you believe the research design is valid. I don’t.”
No, what I think about the research design is irrelevant to this point. You’re simply being sneaky with “effectivness.” If you wanted to argue against effectivness within a study because you say causation can’t be proved, that would be fine. Instead you are pretending effectivness and causation can be somehow separated so you can talk about effect independent of its cause.
“NO. Absolutely not. None of the RCTs I have encountered in my readings have sufficient scientific rigor to demonstrate causation.”
You set an absurdly high standard which, for ethical and practical reasons, can never be met, leaving you free to entertain your hyper-skepticism forever. The same issues affect studies on human psychology and environmental risk factors and, well, there’s a lot of science that can’t afford to live up to your demands. Yet it’s still science and we can still derive conclusions from it by setting up reasonable controls.
Your belief that causation cannot be determined in RCTs is made more absurd by your insistence that, so far as plausibility is concerned, this puts SBM on equal footing with CAM, despite admitting that methodology in CAM trials is worse. Yet, analyses of studies that give priority to data from the more rigorous studies show CAM’s supposed effectivness diminishes sharply the more rigorous the study and goes to no effect beyond placebo for CAM with no scientifically plausible mechanisms (like homeopathy and reiki). Not so in SBM.
“Then we agree at last. A different study design is needed! Thank you!!!!”
Um, pmoran isn’t agreeing with you. He’s talking about a different design being needed for studies meant to discover other things than the efficacy of a particular medicine. You’re still talking about the latter because you seem to think it’s appropriate to set up studies on placebo/nocebo effects within studies for a particular drug. Earlier, you said: “Without a do-nothing baseline group, one has no idea what the placebo effect is, if it is even there.” So pmoran is simply pointing out that this would need to be a study of its own.
“Well, I’m the other fellow. When I read an RCT, I see no evidence that the confounders have been eliminated.”
Because, apparently, you expect all confounders to be addressed by sub-studies within every RCT.
Weing: “I think Dr. Smith can only say “My study showed significant reduction in relapses when compared to a placebo group.” The second statement is not warranted.”
I don’t think the first statement is warranted either.
Reduction implies causation, that the study drug caused the reduction. We don’t know that. We don’t know if some members of the placebo group didn’t suffer psychosocial impacts that triggered relapses, which in turn increased the relapse scores of the entire group. We don’t know that had the data been analyzed without those confounded subjects, there would be no significant reduction in relapses at all between the two groups.
We also don’t know if the opposite happened either and the reduction in relapses would have been much lower. What we know is what the results showed and with what degree of significance. That’s why studies are repeated by others, with different groups of patients, age groups etc. With your approach we shouldn’t even bother studying because we can’t control all the variables. You can still make snowflakes but each one will be different.
A. Noyd; “saying without absolute knowledge, we can’t know which paradigm is more accurate…However, you believe we’re approaching absolute truth at the same time you insist we don’t know which paradigm is getting us there.”
No. We believe our current paradigm is more accurate than our last one. And I believe the next paradigm would more more accurate than the current one. The progress towards absolute truth works because each paradigm corrects the inaccuracies of the last one.
” Instead you are pretending effectivness and causation can be somehow separated so you can talk about effect independent of its cause.”
I am defining the construct of “effectiveness” in a way to allow for practical interpretation without requiring the scientific rigor necessary to establish causation.
People need a drug. We don’t have time or money to do the kind of scientific study that would rule out complex confounders. So we define say, a practical definition of effectiveness that would that allow us to move ahead with strong correlations between drug and outcome. I get that. But I think it is imperative to separate that concept of correlative effectiveness from scientifically established causation.
“You set an absurdly high standard which, for ethical and practical reasons, can never be met, leaving you free to entertain your hyper-skepticism forever.”
They are not absurdly high. They are the minimum required to determine that X caused Y and not A or B. I don’t see how else you can logically determine X caused Y, except by controlling for A and B.
I think medicine sets an absurdly high standard in claiming that an intervention requires scientific proof of causation before using it on patients. So they use a lower non-scientific standard, then relabel the lower standard “science” to appear to be meeting their own criterion.
My argument is this. Call the lower standard exactly what it is: correlational studies unable to establish causation but still useful and important in guiding our choices. And drop the silliness that something as tedious and difficult to establish as scientific proof is needed for medicine. Return the label science to its uncompromised rigor and legitimize medicine without it.
“The same issues affect studies on human psychology and environmental risk factors and, well, there’s a lot of science that can’t afford to live up to your demands. Yet it’s still science and we can still derive conclusions from it by setting up reasonable controls.”
If they cannot live up for the basic requirement of controlling for confounding influences, then they are not science. It doesn’t mean they are not valid and useful and full of reasonable conclusions. Lots of non-scientific research and observation and study is useful and eye-opening and very important. Science is not the only way to know our world.
“Your belief that causation cannot be determined in RCTs is made more absurd by your insistence that, so far as plausibility is concerned, this puts SBM on equal footing with CAM…”
I insisted that you can’t use “plausibility” to rule out CAM right off the bat and dismiss CAM observations without further study. Science has to be more impartial than that. I don’t think I ever said SBM is on equal footing with CAM in terms of plausibility. I know I never said unknown mechanisms have equal plausibility as known mechanisms.
“Um, pmoran isn’t agreeing with you. He’s talking about a different design being needed for studies meant to discover other things than the efficacy of a particular medicine.”
He agrees that a different design is needed to answer those questions. We only disagree whether those questions are necessary to determine efficacy. I made a case that they were.
“Because, apparently, you expect all confounders to be addressed by sub-studies within every RCT.”
At least the obviously huge and major ones. And if it can’t be addressed within the RCT, then at least cite a separate study that does address it. And if you can’t do that, limit your interpretation to correlation-level confidence and do not infer causation from the study.
@ HelenSan, you have demonstrated that medical research confuses you. Do you represent some form of CAM, or do just like to argue?
Joe: “HelenSan, you have demonstrated that medical research confuses you. Do you represent some form of CAM, or do just like to argue?”
How exactly am I confused? I have outlined the definitions of science, the scientific method, experimentation, controls, and confounders in simple and clear terms. I have seen no disagreement with my definitions except that you all seem to think studies that do not control for major confounders can still conclude causation and call themselves “science.”
I do not represent CAM. I am a science enthusiast who reads a lot of journal articles.
Helenscan: “We only disagree whether those questions are necessary to determine efficacy. I made a case that they were.”
Well, no. You are mouthing a vague, confused general disatisfaction with medical research, without giving an example of how any particular “confounder” as not adequately controlled for within the typical efficacy trial.
You should forget that word that you read somewhere and get down to the specifics of how such trials are in error.
Weing: ‘We also don’t know if the opposite happened either and the reduction in relapses would have been much lower.”
That is exactly right. Without controls, we don’t know. And without knowing, without proving, we can’t conclude causation.
“What we know is what the results showed and with what degree of significance. That’s why studies are repeated by others, with different groups of patients, age groups etc. ”
Sure. The results showed a significant correlation between study drug and reduction in relapses. You do more studies and replicate that correlation. I have no problem with that. But we still do not know the influence of confounders and we still cannot conclude causation.
“With your approach we shouldn’t even bother studying because we can’t control all the variables. ”
Not true. Your conclusion assumes that only scientific experiments demonstrating causation are worth doing in medicine. I do not agree with that assumption. Correlational studies and even anecdotal observations (case studies) are very useful, important, and worthwhile. While science chips away at slowly controlling all the variables over decades and approaches causation at a painful snail pace, correlational medical studies like the current RCTs can offer viable options for patients in the here and now.
@HelenSan on 02 Oct 2009 at 5:18 pm “How exactly am I confused? I have outlined the definitions of science, the scientific method, experimentation, controls, and confounders in simple and clear terms.”
As you understand them, very simple.
HelenSan on 02 Oct 2009 at 5:18 pm “I am a science enthusiast who reads a lot of journal articles.”
So am I; but I understand a lot of what I read. I rely on experts when I am not sure.
You still have not looked-up the definition of “sophomoric” here is one “of, relating to, or characteristic of a sophomore; conceited and overconfident of knowledge but poorly informed and immature” http://www.google.com/search?q=define%3Asophomoric&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a
“On the internet, nobody knows if you are a dog” (caption of a famous “New Yorker” cartoon). Maybe you think you are “holding your own” against better educated people than you and, sadly, your husband; but that is just in your imagination.
PMoran: “Well, no. You are mouthing a vague, confused general disatisfaction with medical research, without giving an example of how any particular “confounder” as not adequately controlled for within the typical efficacy trial.”
I did give various examples. Just scroll up. Not controlling for the use of recreational drugs (right after the link to Feynman’s speech), not controlling for the uninterfered natural progression of the disease (to Harriet Hall), the series of unfortunate events in the 10 year trial of 1000 Mr. X’s with gallbladder disease (to Weing), and Dr. Smith’s RCT of study drug X and all the confounders he didn’t control (to Weing).
But ok, I’ll give one more (maybe you don’t read what I write to the others).
Dr. Brown randomly assigns 500 patients with osteoarthritis of the knee to either the study drug X or placebo group in a multi-center study. Outcome measures are the patient’s global assessment of pain in the affected knee and flexion of the affected knee. After 6 months, the study group showed significant improvement over the placebo group in both pain and flexion of the affected knee.
Dr. Brown controlled for diagnosis, frequency and severity of symptoms, medications, age, sex, and race–ensuring that patients were represented equally on these variables in both groups. He concluded that study drug X is efficacious in reducing pain and improving flexion in osteoarthritis of the knee.
How do we know that the study group didn’t have higher scores because it happened to have patients using glucosamine supplements, physical therapy, occupational therapy, or massage therapy? It would have been a simple matter for Dr. Brown to ask, “During the study period, did you use any of the following? If so, how much and how often?” Then Dr. Brown could have reported, “There were no self-reported differences between the two groups in use of physical therapy, supplements, and massage therapy during the study period.” But he didn’t ask, so we don’t know.
How do we know that the placebo group didn’t have lower scores because it happened to have patients who suffered minor injuries during the study period? Maybe some of them moved furniture or exercised too hard and lightly sprained their knees. Again, it would be a simple matter to ask and report that there were no differences in minor injuries during the study period. But he didn’t ask, so we don’t know.
How do we know the study group didn’t have patients that used recreational drugs that altered perceptions of pain? It wouldn’t be the first time someone in chronic pain smoked a joint or scored some heroin. Intermittent drug testing during the study period would have improved the confidence level of the study. But he didn’t try, so we don’t know.
How do we know that the placebo group didn’t happen to have more patients in a geographic location with frequently changing weather, and these patients happened to be sensitive to such changes? Dr. Brown did not control for climate and location of patients, so we don’t know.
You can assume that none of these things happened. But you have no proof that none of these things happened. Without proof, we don’t know that the higher scores in the study group was a result of study drug X and only study drug X. We cannot conclude causation.
@Joe: “Maybe you think you are “holding your own” against better educated people than you and, sadly, your husband; but that is just in your imagination.”
Well, at least I’m not using “You’re a stupid poo-poo head” as an argument.
Helenscan, but all science is about probabilities. This is why statistics are employed. We rarely know for sure when we have reached absolute truth but we can sense when we are close to it, and we can even define that likelihood in very precise terms.
Thus, if 500 patients are properly randomised into two groups it is *highly probable* (but admittedly not *absolutely* certain) with such large numbers that the factors that you are worried about will cancel themselves out.
If the study goes on to produce a highly significant result, statistically, then it is assumed for the practical purposes of medicine that a causal event has probably occurred, because the P level or Odds Ratio or whatever other statistical measure employed could be explained only by a statistically far less likely maldistribution of any important confounder. It can all be worked out.
Actually a single trial is rarely accepted anyway. The FDA normally asks for at least three independent controlled trials before endorsing a drug as effective, partly because the results of any one trial could be due to various chance happenings.
It is also usual in quality research these days to perform a post hoc comparison of groups for known or possible confounders as a check on the randomisation.
There is a germ of truth in what you say, but statistical analyses and average scientitific rigour can cope with the matters you raise.
Pmoran: “Thus, if 500 patients are properly randomised into two groups it is *highly probable* (but admittedly not *absolutely* certain) with such large numbers that the factors that you are worried about will cancel themselves out.”
Oh yes, I forgot. Big Sample and Random, the confounder elves.
(Just ribbing ya.)
All our knowledge is tentative. We are not as close to absolute truth as physicists. Neither are chemists for that matter. We can get into a peeing contest as to whose Kung Fu is more powerful, but I don’t think it will get us anyware.
It appears you are looking down at the use of sampling in medical studies and the errors that are inherent despite randomizing. But physicists also use sampling. If you have a kilogram of a radioactive isotope that has a half life of 10 years, how do you know all the atoms in your sample aren’t exactly those that are ready to fission in the next 10 minutes? Ah yes, Big Sample and Random. Good for the goose but not for the gander?
@Pmoran
I have a question. If it is highly probable that you can trust the Confounder Elves (Big Sample and Random) to cancel out confounders, then why not trust them to cancel out ALL the confounders? Why not trust them to cancel out any significant differences in medication use, age, sex, and race as well? Why do any controls at all?
Weing: “Good for the goose but not for the gander?”
Physicists do not use Big Sample and Random INSTEAD of controls. If selection bias of atoms that are “ready to fission” is a concern, they will measure it and make sure that is equally represented in both the study sample and the control sample.
Big Sample and Random supplement controls for confounders. They do not replace them. If controls are the actors on a stage, Big Sample and Random are the backstage staff. You can’t try to save money on hiring actors by using backstage staff in the main roles of a play. Well, you can, but the play won’t be very good.
@Pmoran, another question.
What if I told you that in Dr. Brown’s RCT, the study drug was actually a CAM intervention? Would you still think the study demonstrated causation and the uncontrolled confounders were likely to have been canceled out by Big Sample and Random?
@HelenSan on 03 Oct 2009 at 11:44 am
Besides sophomoric, you should look-up “naive”. Really.
If a CAM intervention were ever supported by high-quality research (as described by pmoran and others, not you nor you hubby)- we would accept it and call it medicine.
“If selection bias of atoms that are “ready to fission” is a concern, they will measure it and make sure that is equally represented in both the study sample and the control sample.”
Why would selection bias of atoms that are ready to fission be a concern? I’m not a physicist, but is that a concern of nuclear physicists? I thought the decay rate was similar to the death rate. As with atoms so with people. You can know that one person dies of melanoma every hour in the US but you cannot tell which one is next.
weing on 03 Oct 2009 at 2:27 pm
“You can’t use reason to talk someone out of a position they didn’t use reason to arrive at.” PZ Myers
Joe,
I’m not trying to talk her out of her position. Just trying to understand. Anyway, reason is the only thing I’ve got.
Weing,
Her position is that she has a juvenile knowledge of science and she won’t be swayed from it. That is all one needs to understand. You use reason well, HelenSan and hubby (PhD!?) are simply impervious.
When I look at graduate (PhD) students today (in chemistry), they bring tears to my eyes compared to what they were before WW2. After that time, the government stepped in and the degree was open to anyone who was willing to work for a low wage for a few years.
We are dealing with people with an inflated view of their knowledge and understanding.
Weing: “If you have a kilogram of a radioactive isotope that has a half life of 10 years, how do you know all the atoms in your sample aren’t exactly those that are ready to fission in the next 10 minutes?….Why would selection bias of atoms that are ready to fission be a concern? I’m not a physicist, but is that a concern of nuclear physicists?”
Well, the answer is no. Selection bias of atoms that are “ready to fission” is not a concern to any physicists I know.
When you asked, “how do you know all the atoms in your sample aren’t ….X…?” you were asking a question about selection bias. My answer was, no matter what X might be (ready to fission as you say or something else), if physicists are concerned that they have selection bias of X in the sample, then physicists measure X and control for it. They don’t leave it to Big Sample and Random to magically cancel out.
Incidentally, what is considered a large sample size to physicists is very different than what it is to physicians. A kilogram of uranium isotope will have roughly 2.5 x 10^24th atoms. That kind of sample size can decrease the probability of selection bias in a significant way that a sample size of 500 can’t. In this context, very few medical studies actually have large sample sizes.
But the confounders I am talking about goes beyond selection bias. You may have two equally substance-abuse-free groups at the outset of the study. No selection bias. But that doesn’t guarantee that patients won’t start drinking and smoking joints during the study period, inserting a substance abuse bias in one group more than the other. Bias doesn’t occur only at the beginning of a study. It can occur at any time. That is why a good research design needs to control for them–at the beginning and throughout the study.
Large sample sizes and randomization can only decrease the probability of selection bias. They are not controls and no other science besides medicine uses them INSTEAD of controls.
Weing: “Anyway, reason is the only thing I’ve got.”
Good to hear Weing. I would be curious about your answer to the question I posed to Pmoran. If I told you Dr. Brown’s study drug was actually homeopathic remedy X, but the rest of the study was exactly as described, would you say the RCT proved that homeopathic remedy X to be effective over placebo?
This is very much a question of logic and reason. Same study, same results–different substance used as the independent variable. I want to know if what works for the goose works for the gander.
Helenscan: “I have a question. If it is highly probable that you can trust the Confounder Elves (Big Sample and Random) to cancel out confounders, then why not trust them to cancel out ALL the confounders? Why not trust them to cancel out any significant differences in medication use, age, sex, and race as well? ”
Yes, indeed, why not? Just ensure numbers are sufficiently large and truly allocated at random.
The question actually makes little sense because if we are performing a study to find out whether a treatment ever works or not , we are hardly likely to yet know all possible confounders. We have to rely upon a study design that makes chance maldistribution of confounded subjects a much less likely happening than the causal influence being sought.
.
Helenscan:” @Pmoran, another question.
What if I told you that in Dr. Brown’s RCT, the study drug was actually a CAM intervention? Would you still think the study demonstrated causation and the uncontrolled confounders were likely to have been canceled out by Big Sample and Random?”
I think you are obsessing about confounders, when they are merely one of a very large number of potential errors in the design, performance and interpretation of clinical studies.
Loss of blinding and consciously or unconsciously swaying the results by finding excuses for exclusion of a few patients would be more common flaws. Many of the possible errors cannot be detected from the published material.
If the CAM method you mention above was a herb and the effect plausible, then I would accept a certan level of likelihood of causality, pending replication in other studies.
If it was a remedy depending upon homeopathic dilutions, I would find it very difficult to take a result showing it working better than placebo seriously.
That is, again, all to do with the probabilities. The likelihood of study error is much higher than that homeopathic remedies could have intrinsic therapeutic activity. A lot of different lines of evidence go into that call.
Pmoran: “If the CAM method you mention above was a herb and the effect plausible, then I would accept a certan level of likelihood of causality, pending replication in other studies.
If it was a remedy depending upon homeopathic dilutions, I would find it very difficult to take a result showing it working better than placebo seriously. ”
=====
I think this brings us full circle to the beginning of this thread, when Wondering was arguing, “You can’t ignore RCTs and statistics merely because you don’t agree with them.”
Here we have same methodology, same design, same results. Then depending on what study drug X actually is, you will give this same study different interpretations. For a pharmacological product, the study demonstrates causation, confounders are canceled out. For an herbal CAM, the study demonstrates likelihood of causation, confounders are probably canceled out. For a homeopathic remedy, the study demonstrates nothing at all; the results must be caused by confounders or outright fraud.
Thank you for being honest about your interpretation. This is a common answer I’ve gotten from “S”-BM proponents and “skeptics”: a double or triple standard for interpretation of the same exact “scientific” evidence. For RCT results we agree with, we accept wholeheartedly. For RCT results we like less, we accept with reservations. For RCT results we dislike and find “implausible,” we absolutely will not accept no matter what; you might as well not bother doing them. What is good for the gander will never be good enough for the goose.
And the point Wondering made, and what I am trying to say throughout all this, is: Science is objective, that is to say, blind to personal opinion. It picks a standard, then applies that standard equally to all evidence no matter what it is for. The skepticism of science is uniformly applied; it does not depend on the true identity of “study drug X.”
If the methodology and sample size were right, then I could not ignore the results of the study and would be forced to accept them. I would still have to wonder, as with any pharmaceutical studies, about what role the file drawer effect was playing. Would I want to see the study replicated? Of course, I would. Would your hubby want to replicate a study that showed a kilo of Uranium 232, after treatment with a homeopathic solution, changed its half life to 10 days?
Helenscan: — OUR double standard?. We have just been discussing a few of the many ways in which clinical trials can give spurious results, yet you now chastise us for not trusting your hypothetical study sufficiently?
I even indicated that a single study is never enough for mainstream endorsement.
We now know that drug companies have been able to somehow “obtain” positive results for some of their products, so that we now distrust their offerings to some extent. For a long time no one trusted clinical trials from China as their journals NEVER published negative results, no matter what was being tested.
Is this reflecting a double standard, or simply an indication that context and the biases of researchers inevitably feed into the trustworthiness of research — before you even start to consider the extreme implausibility of methods such as homeopathy?
The clinical trial is a rule-of-thumb work-around that, despite its vulnerability to bias and error, does work effectively over time within the medical system that evolved it. The sheer volume of research performed upon any important treatment method ensursx that useless methods don’t last long. It works best with pharmaceuticals, and physical outcomes, and less well with procedures, lifestyle measures and subjective outcomes, so that some questions can remain a matter of controversy through inconsistent results for years.
Our experience of this system allows us to predict that inactive treatment methods will produce some positive results over placebo, especially if the work is conducted by enthusiasts seeking vindication and the work also contains various indicators of low quality research . Publication bias will also tend to exaggerate the number of positive vs negative ones.
It is an admittedly tough road for implausible methods to follow. They could gain some credibility if they were able to show clear cut effects upon important, objective independently measurable disease outcomes, or produce desperately needed basic research into core claims such as “like cures like” instead of retreating into mysticism when such notions are challenged.
@Weing
I am very impressed with your answer. As you can see, Pmoran (and many other “S”BM supporters and “skeptics” I have talked to) disagree with you about being forced to accept the results. It is a pleasure to meet one who can apply standards objectively.
Weing: “Would I want to see the study replicated?”
Lots and lots of replication goes without saying.
Incidentally, since you asked about what my husband would want, he says he cannot publish at all until he can get 100% correlation between the independent and dependent variables in the study group, and 0% correlation in the control group. That means, he has to adjust his controls, in tedious experiment after experiment, until he has eliminated all the confounders that influence the measured outcome. In the end, if everything is controlled for and equal, he ALWAYS gets the measured outcome every single time he applies the independent variable, and NEVER gets the measured outcome when he doesn’t. And then he has to replicated the study a minimum of 3 times.
Then and only then, he has a publishable paper. And even then, he can only conclude, “It is LIKELY that this experiment demonstrates causation.”
All the correlations he observes until he gets to that point? They are the basis for further hypothesis and experiment–not the basis for arriving at a conclusion. He said if he ever tried to publish a correlation like they do in medicine (where the outcome is not obtained 100% of the time), he would be laughed out of work.
(And for background info, he has been a full time research scientist for 7 years. That means for 7 years, he has been observing, hypothesizing, experimenting, publishing, and presenting for a living. In these 7 years, he has published over 50 papers, not including presentations. Each and every single one of these papers showed a 100% correlation. I say all this so you understand where he is coming from.)
—-
Now I understand completely that medicine cannot achieve this level of scientific rigor. They do the best experimentation they can, with what constraints they got. That is not a problem.
My issues are these:
1. They should not infer causation from correlation. And every correlation that is less than 100% in the study group, and more than 0% in the control group, is just a correlation. It means other factors are still influencing the measured outcome that they are not controlling for–and they cannot attribute cause until those factors have been eliminated in the research design.
2. They should carefully qualify their conclusions to match their data. They need to say at the end of each and every RCT, “What we have is a strong correlation between X and Y. We did not control for confounders A, B, C, D, and E–all of which could have influenced Y. We did not replicate. Future research needs to replicate this study and address confounders.”
3. “Proof of effectiveness” needs a standard operating definition that can be obtained from correlational studies (since true experiments are quite impossible in most cases). Make it a legal definition of effectiveness if you wish. Then apply that standard equally and objectively no matter what the intervention is, whether physicians like the intervention personally or not.
4. Rename this website “Legal Proof-Based” Medicine or “Effectiveness”-Based Medicine, and then clearly and objectively define what “Legal Proof” or “Effectiveness” means.
5. Stop lowering the standards of scientific rigor so that medical correlations with innumerable shortcomings can jump onto the the “science” bandwagon. With ethical constraints, medicine cannot do true experiments, medicine cannot do true controls, so medicine is not a science and shouldn’t be one. Live with it. Legitimize yourself in some other way. Leave scientific rigor up high where it ought to be and just openly admit medicine doesn’t have any scientific rigor and never could because you guys are ethical and decent people.”
pmoran: “OUR double standard?. We have just been discussing a few of the many ways in which clinical trials can give spurious results, yet you now chastise us for not trusting your hypothetical study sufficiently?”
The double standard exists because you trusted the hypothetical study sufficiently when “study drug X” was something you liked, and distrusted it when “study drug X” was something you didn’t like. Same study, same design, same methodology, and same results. Yet you would interpret that same study differently based on the true identity of “study drug X.”
Pmoran: “They could gain some credibility if they were able to show clear cut effects upon important, objective independently measurable disease outcomes,…”
What would you call “clear cut effects”? Because Dr. Brown’s study, with both subjective and objective outcome measures, showed “clear cut effects” to the point you would say it demonstrated causation. Yet if it were shown with a homeopathic remedy, you would reject it as “clear cut effects.”
So what kind of RCT would you accept for homeopathy?
@ HelenSan on 04 Oct 2009 at 10:45 am “So what kind of RCT would you accept for homeopathy?”
Aren’t you going to tell us? After all, you know so much more than we do.
HelenSan: “No. We believe our current paradigm is more accurate than our last one. And I believe the next paradigm would more more accurate than the current one. The progress towards absolute truth works because each paradigm corrects the inaccuracies of the last one.”
Hm, yet you later say: “I insisted that you can’t use ‘plausibility’ to rule out CAM right off the bat and dismiss CAM observations without further study. Science has to be more impartial than that.”
Noting first that your portrayal of us as dismissing CAM “without further study” is a strawman (you’re confusing expectations with conclusions), these two just do not go together in reality. You are essentially saying that, despite your belief that our current “paradigm” is more advanced, that we cannot rely on that to judge the value of taking utterly implausible CAM seriously and must remain eternally and equally credulous or we’re not “impartial” enough. Bull. You simply do not understand science, and this here illustrates it perfectly. This is the sort of facile thinking that allows people to believe, for instance, we must take intelligent design as seriously as we take evolution.
“I am defining the construct of ‘effectiveness’ in a way to allow for practical interpretation without requiring the scientific rigor necessary to establish causation.”
Here’s more postmodern relativistic weasel language, by the way. I understand what you’re trying to do, but it’s silly. Either some effect Y is caused by X or it’s not. Either effectiveness is established or it’s not. Using a “practical interpretation” of a “construct” is just confusing, especially when your argument revolves around our supposed lack of proof of causation. So you ought to stick to convention here.
“They are not absurdly high. They are the minimum required to determine that X caused Y and not A or B. I don’t see how else you can logically determine X caused Y, except by controlling for A and B.”
Goooo, argument from incredulity. They are absurdly high and they’re that way because you’re a denialist. Or vice versa–definitely a correlation here!
“Return the label science to its uncompromised rigor and legitimize medicine without it.”
How about we just use science the way scientists use it (and your husband is too small a sample size to stick to his opinion) and not the silly, warped misconception you cling to in order to feed your denial.
“Science is not the only way to know our world.”
Oh, reaaaaally? I thought you weren’t a relativist. Or maybe you’re the first to come up with a coherent epistemology for knowing our world outside of methodological naturalism.
“At least the obviously huge and major ones. And if it can’t be addressed within the RCT, then at least cite a separate study that does address it. And if you can’t do that, limit your interpretation to correlation-level confidence and do not infer causation from the study.”
In other words, do what’s already being done.
It’s been clear to me for a while that you’re a denialist. You use what you think are reasonable objections in order to maintain your disbelief, but your understanding of science and how it is practiced in fields like medicine is highly lacking. In order to do medical research your way, we’d need several undesirable or impossible things, such as untreated control groups or keeping watch on every study participant 24 hours per day or the ability to perfectly replicate conditions between trials. Since those are never going to happen, you get to cling to your denial of causation in medicine forever. It’s exactly like the evolution denialists who want to see a transitional fossil linking every species there ever was before they’ll accept evolution is true, insisting, till that happens, we put warning labels in textbooks about how “evolution is only a theory.”
“This is a common answer I’ve gotten from ‘S’-BM proponents and ‘skeptics’: a double or triple standard for interpretation of the same exact ‘scientific’ evidence.”
It’s not a double standard. After all, pmoran isn’t saying he’d judge your study conclusive either way. In fact, I distinctly recall him saying that the FDA requires three or more studies before any talk of causation is allowed and he seems to agree with this. However, it’s entirely reasonable to expect that studies showing a positive effect for CAM treatments with an implausible mechanism would be cancelled out by later studies. And, so far, reality has always lived up to our expectations–more high quality studies do, indeed, fail to show a causative effect for implausible CAM. This is not the case for the majority of conventional treatments or a significant percentage of CAM with plausible mechanisms.
“For RCT results we agree with, we accept wholeheartedly. For RCT results we like less, we accept with reservations. For RCT results we dislike and find ‘implausible,’ we absolutely will not accept no matter what; you might as well not bother doing them.”
This is only your strawman version of how things work. We do not discard or discount results merely on whim, no one bases effectiveness on a single study, and if CAM ever shows result to the same degree that SBM does (in multiple, high quality studies!), it gets accepted. Your reading comprehension is abysmal if you can still cling to these ideas after so many comments. Nothing gets through to you, so there’s no point in arguing further. Respond if you wish, but I’m going to sit back and laugh at you now.
pmoran: “If the study goes on to produce a highly significant result, statistically, then it is assumed for the practical purposes of medicine that a causal event has probably occurred, because the P level or Odds Ratio or whatever other statistical measure employed could be explained only by a statistically far less likely maldistribution of any important confounder. It can all be worked out.”
See, now, that’s what it looks like when someone knows what he’s talking about.
@A. Noyd,
You got it.
pmoran: “If the study goes on to produce a highly significant result, statistically, then it is assumed for the practical purposes of medicine that a causal event has probably occurred, because the P level or Odds Ratio or whatever other statistical measure employed could be explained only by a statistically far less likely maldistribution of any important confounder. It can all be worked out.”
A. Noyd: “See, now, that’s what it looks like when someone knows what he’s talking about.”
====
And that is what I call statistical legerdemain, or sleight-of-hand, used to make inferences of causation without the actual data in hand. It might work on lay people and physicians, but it doesn’t work on anyone who understands statistics.
A. Noyd: “You are essentially saying that, despite your belief that our current “paradigm” is more advanced, …”
More advanced than the last paradigm. Not more advanced than future paradigms.
“…that we cannot rely on that to judge the value of taking utterly implausible CAM seriously and must remain eternally and equally credulous or we’re not “impartial” enough.”
Because future, more accurate paradigms might uncover natural mechanisms through yet undiscovered technology that explain CAM and make them more plausible. We don’t have to be credulous to be impartial. We just have to be open to further investigation of implausible anomalies.
I think I have beat the paradigm and relativism horse enough now. If you still refuse to understand my position, so be it.
@HelenSan on 04 Oct 2009 at 2:54 pm “… If you still refuse to understand my position, so be it.”
We understand your position and recognize it as ignorant nonsense, so be it.
@ HelenSan on 04 Oct 2009 at 10:45 am “So what kind of RCT would you accept for homeopathy?”
Aren’t you going to tell us? After all, you imagine you know so much more than we do.
HelenSan,
I’m curious. Do you accept that smoking cigarettes can cause lung cancer? There are no RCTs of the kind you say are necessary to establish a causal relationship (ruling out all possible confounders), but there is an overwhelming amount of “circumstantial” evidence from various other sources to support the hypothesis. And all the evidence is coherent, with a known mechanism, lab and animal studies, a dose-response relationship, etc.
“Because future, more accurate paradigms might uncover natural mechanisms through yet undiscovered technology that explain CAM and make them more plausible.”
I was not aware that CAM has not been already explained. More accurate paradigms have made CAM less plausible. In the old days when it was thought a pneumonia was caused by punishment by gods, it was plausible that sacrificing a goat might lead to recovery.
“We just have to be open to further investigation of implausible anomalies. ”
What implausible anomalies do you have in mind?
hELENSCAN said: “Proof of effectiveness” needs a standard operating definition that can be obtained from correlational studies (since true experiments are quite impossible in most cases). Make it a legal definition of effectiveness if you wish. Then apply that standard equally and objectively no matter what the intervention is, whether physicians like the intervention personally or not.
4. Rename this website “Legal Proof-Based” Medicine or “Effectiveness”-Based Medicine, and then clearly and objectively define what “Legal Proof” or “Effectiveness” means. ”
PM > What you ask is a practical impossibility. There are not merely double standards in acceptable levels of evidence within the sciences, there are multiple standards. And as pointed out before, we will never have one that delivers confidence in absolute truth.
How come? Well, I hold that in medicine and other practical sciences “adequacy of evidence” ALWAYS carries with it an unspoken “for this particular practical purpose.” You cannot define the former without knowing the latter and sometimes also taking into account practicalities that can limit what evidence is obtainable, as in Harriet’s example.
Some examples of the practical implications of evidence levels—.
.
- The probability of failure of a jet engine rotor blade MUST be less than the likelihood of the cockpit door malfunctioning.
- A physicist’s work is not accepted as valid without reaching a statistical significance of one in some thousands (p< .001 from memory), whereas a medico only needs one of about one in twenty to be claiming a "positive" result, with the limited implications of that.
- A doctor has a patient with a problem for which nothing is working, He comes across a study showing that drug X or herb Y works better than placebo at a statistical significance of p<.05. He may quite reasonably choose to try that treatment on his patient, even knowing (As A Noyd points out) that subsequent trials may very well contradict that finding.
- In our recent argument with chiropractors concerning the risk of stroke from neck manipulation we pointed out that a few anecdotal reports are enough to get any serious potential risk listed on the warning label of a drug, even though far tighter evidence is demanded for efficacy.
Now, apply that understanding to a "positive" clinical study of an ultra-diluted, "like cures like", symptom-focused, "memory of water" homeopathic remedy that has been dripped onto a tiny pillule and allowed to dry out.
Other things being equal, a single study like this has the practical outcome of slightly to moderately elevating the probability that a treatment works better than placebo, to a degree depending on many factors, some of which we have discussed .
But other things are NOT equal in this instance. Whether you yet see it or not, the correct scientific response to such a result is "WTF —?" , because we cannot accept ANY likelihood that such a remedy is active without also having to accept that the five or six extremely unlikely processes that are required to make it work are not only very likely to be valid, but also work reliably enough to form the basis of a whole system of medicine.
This is a heavy burden. And there is a lot of directly contradictory evidence for those processes. They would require multiple studies to at least physicist's level of statistical significance before being regarded as valid (close to truth). It is a corruption of the scientific method to even dream of using fragile clinical studies as a surrogate for such desperately needed basic research (not that such research has not been tried many, many times without success).
Cannot you see how such a study automatically takes on a breathtaking level of practical significance, one going far, far beyond whether a little pill works better than placebo for a subjective complaint for which there may be other just as effective treatments? You cannot ignore the ancillary questions that the study raises, and pretend that "but this is how the mainstream works".
Yes, if it was a drug a lot of doctors might take a gamble on it. Some even try homeopathy, not caring much if it is just a placebo., Trying out a medical treatment does not require anywhere near the same level of "proof" as changing fundamental understandings of the universe or the nature of therapeutics.
We skeptics have created this "double standard" furphy by previously harping on about the "Gold Standard" of RCTs, not realising that they do their job within the mainstream only through the presence of a lot of other checks and balances, and a seamless integration with the basic sciences and technology..
Harriet Hall: “I’m curious. Do you accept that smoking cigarettes can cause lung cancer? ”
I accept that there is a correlation between chronic cigarette smoke and lung cancer, and that this correlation may represent a causal relationship. But then again, it may not. I know animal studies have shown inconsistent results, which the truth be told, I have not examined in great detail.
The first thing I would have to do to answer your question with more conviction is to evaluate the quality of the animal studies vis a vis their results. If I find a high degree of correlation between smoking and lung cancer in the higher quality animal studies, replicated frequently, I would accept that smoking is probably carcinogenic to certain animals, and likely to be carcinogenic to humans as well. If I don’t find good quality animal evidence, then I would be much more undecided.
I actually find a lot of similarities between smoking and lung cancer and mercury and autism, though the latter has a smaller body of evidence because the controversy is younger. There are correlations, some better researched than others. The mechanism is, ahem, plausible. The biggest similarity for me is the jumping to conclusions about one factor being the sole cause of a complex diagnosis with multiple etiologies.
So even if the correlations are very high, I see smoking “causing” lung cancer like sexual intercourse “causing” babies. It is not necessary, nor sufficient. A lot of factors that have to be just right for it to serve as a causal factor among many.
@ HelenSan on 04 Oct 2009 at 10:45 am “So what kind of RCT would you accept for homeopathy?”
Aren’t you going to tell us? After all, you imagine you know so much more than we do.
No Helen, a future “paradigm” does not change the data. We already have enough data about CAM to know that CAM works no better than a placebo. Maybe if the people for whom CAM doesn’t work could be excluded and only a highly susceptible subset treated with CAM, then CAM might look better in that subgroup but that doesn’t change the data that CAM is ineffective in the larger group.
I wrote the above before I saw your comment on autism and mercury where you compare it favorably to the epidemiology of smoking and cancer. You have just shown yourself to be an ignorant hack. There is essentially no plausibility for mercury to cause autism. They symptoms are different, there is no excess of autism in regions with high mercury exposure (look at the Faroe Islands where they have mercury levels more than an order of magnitude higher than in the US and there is no excess autism. The most characteristic physical symptom of autism is the number of minicolumns in the brain and that is fixed at 6-8 weeks in utero. Nothing 2 years later can change that.
Mercury exposures 70 years ago were much higher than they are now. Mercury poisoning due to mercury containing teething powders were a leading cause of death in children. Many tens of thousands of children were sickened by mercury poisoning (called pink disease) and over a thousand children died from the mercury in teething powder. Many tens of millions of children each received many thousands of times more mercury from teething powder than any child has received from vaccines. Many teething powders contained over 55,000 micrograms of mercury per dose and children were given multiple doses multiple times as their teeth came in.
If you think mercury is a credible cause of autism then you have no critical thinking skills at all. pwned.
Daedulus: “No Helen, a future “paradigm” does not change the data. We already have enough data about CAM to know that CAM works no better than a placebo.”
In those particular studies. The file drawer effect exists for many kinds of intervention. There are always negative studies that remain unpublished. Pointing to the ones that do get published as evidence of absence, while ignoring the ones that remain successfully hidden….eh…doesn’t seem very consistent to me. The question is, is the sample of those particular negative studies representative of all studies on the subject? And even if it is, does that exclude that future studies with new instrumentation and technology and research designs will find positive effects?
For example, take studies on homeopathy. There are over 3000 possible remedies to choose from. According to proponents, the right one has to be prescribed to fit a particular individual, not diagnosis. Two persons with the same diagnosis often get different remedies. Choosing the right one for each individual is heavily dependent on the interviewing and intuitive knowledge of the homeopath. The wrong one will do nothing. Many RCTs use the same remedy for all patients, which is contrary to how homeopathy is supposed to be practiced.
So a good RCT design for homeopathy would use the same provider for both all the patients, in both the placebo and study group. That would control for the quality of skill and service involved in choosing the “right” remedy out of 3000. Double-blindness would apply of course. This is an example of how a different RCT design might reveal results not observed in other studies.
“They symptoms are different, there is no excess of autism in regions with high mercury exposure (look at the Faroe Islands where they have mercury levels more than an order of magnitude higher than in the US and there is no excess autism.”
I would love to read this study. Would you happen to have the full text? If so, you can email it to me at helensan@att.net. If not, do you have the complete citation?
” The most characteristic physical symptom of autism is the number of minicolumns in the brain and that is fixed at 6-8 weeks in utero. Nothing 2 years later can change that.”
ASD/PDD is a heterogeneous disorder with multiple etiologies. It really has become a wastebasket diagnosis for many children who do not have classic Kanner autism or even classic autistic symptoms besides speech delays and some inability to socialize. The characteristic physical symptoms and biomarkers vary greatly within this group. It is entirely possible for a subgroup of patients with the ASD/PDD diagnosis to be exhibiting symptoms of mercury poisoning while being labeled, “autistic.”
“If you think mercury is a credible cause of autism then you have no critical thinking skills at all. ”
I said mercury and autism have similarities to smoking and lung cancer in the type of evidence provided to prove causation. At the moment, I find imperfect correlations in both controversies and no proof to demonstrate causation in humans. (Though like I said, I haven’t looked closely at the animal studies in smoking–and that may tip my views.) That is a far cry from finding mercury “credible” as a cause for autism. If one wishes to exaggerate, I would prefer you exaggerate in the other direction, that I find both “incredible,” if you will.
HelenSan on 05 Oct 2009 at 10:07 am “I said mercury and autism have similarities to smoking and lung cancer in the type of evidence provided to prove causation. At the moment, I find imperfect correlations in both controversies and no proof to demonstrate causation in humans.”
There you go- true nonsense wrapped around (maybe) a bit of truth. Both topics rely, in part, on epidemiological data; but the best data implicates tobacco smoke and vindicates mercury.
Are you the same HelenSan who posted in favor of using homeopathy, based on anecdote, at the JREF forum 2 years ago. If so, your critical thinking skills are demonstrated, once again, to be lacking.
http://forums.randi.org/showthread.php?p=5172102#post5172102
HelenSan,
I am not a party to the later part of this long standing debate, but your last post brought a few things to mind:
3000 possible ‘remedies’ of what? For each symptom? Each disease? Each individual? Therefore, in a country of, say, 1.4 billion people, there are potentially 1.4 billion types of ‘remedies’ possible, and your homeopath decides on the right one by ‘intuition’?
And how does that remedy work? No doubt you are well aware of the homeopathic principle of very large dilutions, often to the range of leaving virtually no molecule of the drug in the solution. A 6X dose reflects 10^6 times serial dilution, which probably is not so bad as far as the presence of the drug is concerned, but it is supposed to increase in ‘potency’ with further and further dilutions, which is a physical impossibility. Homeopaths often prescribe 30X and 200X doses, and then they ‘explain’ the idiocy by randomly pulling out untenable concepts like water memory!
Since you talk about ‘paradigms’ and all, HelenSan, what is the scientific basis of the homeopaths’ insistence on the mind-body ‘axis’ for virtually every disease, including infectious ones? I am sure you know that this is why the homeopathic medicines are ‘personalized’ for individual patients based on their ‘personality’ and ‘history’, when neither has anything to do with the bug – in case of an infectious disease.
In the name of ‘practicing medicine’, homeopaths represent a gross ignorance of physiology in health and disease, as well as pathogenesis.
ARE YOU FOR REAL? Holy Crrr…. *faints*
Kausik Datta: “3000 possible ‘remedies’ of what? For each symptom? Each disease? Each individual? ”
Remedies are made from succussed (shaken) dilutions of substances. The 3000 remedies represent dilutions of 3000 substances. Each remedy is supposed to “cause” a pattern of symptoms if given to a healthy individual. If your symptom pattern matches that of a specific remedy, that remedy is considered to be a possible candidate for triggering healing in you. The problem is, many of the symptom patterns overlap, so it is extremely difficult to match the right pattern to the right patient. That is why a good intuitive grasp of both the remedy pattern and the patient’s pattern is essential for singling out the “correct” remedy. That is why the “successful” practice of homeopathy is highly dependent on the skill of the provider.
“Since you talk about ‘paradigms’ and all, HelenSan, what is the scientific basis of the homeopaths’ insistence on the mind-body ‘axis’ for virtually every disease, including infectious ones?”
I don’t know what “mind-body axis” you’re talking about. To date, there is no known mechanism meeting scientific rigor that would explain how homeopathy is supposed to work.
@Joe: “Are you the same HelenSan who posted in favor of using homeopathy, based on anecdote, at the JREF forum 2 years ago. ”
Yes, I am that same HelenSan. Except I was not advocating the use of homeopathy, and certainly I was not advocating anything based on one anecdote. I was advocating that there is nothing wrong with patients wanting to try homeopathy (or any CAM) based on observed temporal correlations between remedy and desired outcome. It is the same thing I have advocated here.
HelenSan:
Nothing wrong?
HelenSan on 05 Oct 2009 at 11:54 am “… I was not advocating the use of homeopathy”
HelenSan at JREF “I use homeopathy because I have observed empirically that there is a temporal correlation between using a remedy and being relieved of symptoms. … It is very cheap, and it has virtually no side effects. If my symptoms disappear and don’t come back, what’s wrong with that?”
http://forums.randi.org/showthread.php?p=3282149#post3282149
What’s wrong is that you demonstrate lack of critical thinking. Homeopathy is a sham and anecdote has little value to a scientist.
Helensan wrote:
That rather depends on what you mean by “wrong.” I agree it’s not wrong in the sense that it shouldn’t be illegal or anything. (At least, not illegal for the patients. The providers are arguably another story, IMO.)
But in the sense of wanting a treatment that’s likely to be effective by any means other than placebo effect, then yes, it is wrong.
I also note that you carefully avoided commenting on the dilution principle of homeopathy. Since there is nothing except water (or sugar globules) in the concoction that the patient is getting, it makes your point about 3000 or 30000 remedies quite moot, doesn’t it?
And you think it is still okay for patients to keep taking this quack nostrum on faith, is that it?
Pmoran: “Well, I hold that in medicine and other practical sciences “adequacy of evidence” ALWAYS carries with it an unspoken “for this particular practical purpose.” ”
That is actually a very good point, with which I agree wholeheartedly. Where I depart is while the evidence is adequate for a particular purpose, if it does not have scientific rigor, don’t call it “scientific” evidence. Just call it “adequate” evidence. Define what “adequate” means and apply it equally to all studies meeting those criterion.
“because we cannot accept ANY likelihood that such a remedy is active without also having to accept that the five or six extremely unlikely processes that are required to make it work are not only very likely to be valid, but also work reliably enough to form the basis of a whole system of medicine.”
I understand. If A is unlikely to happen, and B is unlikely to happen, then C has a near zero chance of happening. But what if C functions independently of A or B?
Let’s go back to homeopathy. It is assumed that remedies cannot work because they have no pharmacologically active ingredients. But what if homeopathy works electromagnetically rather than pharmacologically? Enzymes are known to change activity and functions in different pH and temperature environments, for example. What if miniscule electromagnetic influences affect their functions as well? If such mechanisms are discovered in the future, homeopathy may not seem as unlikely as it does now.
Now I can go into all sorts of sci-fi-like speculation about homeopathy might be possible. But that is not the point I am trying to make. My point is that “likelihood” is an opinion based on certain assumptions and premises. If those assumptions are incomplete in any way, things considered unlikely now may be considered more likely with additional information.
So it boils down to this: people who believe their current knowledge systems to be nearly complete are more likely to believe that CAM effects cannot possibly exist. People who believe the current knowledge system to be vastly incomplete are more likely to be open to discovering new mechanisms for CAM. Obviously, you are one of the former, and I, the latter.
Despite that, we both share a commitment to science and objective evidence. So let’s put opinions of probability aside and just look at the evidence. If Dr. Brown’s RCT demonstrates evidence that Drug X or Remedy X shows significant improvement over placebo, for the purposes of alleviating the suffering of osteoarthritic patients, and it can be replicated, then why not apply that evidence and allow patients to try it with informed consent? Why not advocate or even just allow for further investigation by science?
” It is a corruption of the scientific method to even dream of using fragile clinical studies as a surrogate for such desperately needed basic research (not that such research has not been tried many, many times without success).”
No one here, esp not I, is advocating using clinical studies as a surrogate for basic research. Use clinical studies for what they were design: to determine whether or not patients should try X intervention.
“Yes, if it was a drug a lot of doctors might take a gamble on it. Some even try homeopathy, not caring much if it is just a placebo., Trying out a medical treatment does not require anywhere near the same level of “proof” as changing fundamental understandings of the universe or the nature of therapeutics.”
No argument from me here. I agree completely. Again, my only issues are: 1) calling the lower level of proof “scientific” and 2) expecting a higher level of proof from CAM before allowing patients to try it, which defeats the purpose of the clinical study.
“We skeptics have created this “double standard” furphy by previously harping on about the “Gold Standard” of RCTs, not realising that they do their job within the mainstream only through the presence of a lot of other checks and balances, and a seamless integration with the basic sciences and technology.”
Well said! But it doesn’t change the fact that if CAM meets the “Gold Standard,” even without this seamless integration, it DID meet the Gold Standard. That alone should count for something. So demand more replication. Demand more mechanistic studies. Stay skeptical. That’s all good. Pronouncing it impossible and throwing the study out as if it never happened–THAT is not good. That is not good science.
HelenSan: “But what if homeopathy works electromagnetically rather than pharmacologically?”
Hahahahahahahahaha *snort* hahahahahahahaha *wheeze* hahahahahahaha!
“…people who believe their current knowledge systems to be…”
(Psst, no one’s fooled if you substitute “current knowledge system” for “paradigm,” you know.)
Kausik Datt: “Since there is nothing except water (or sugar globules) in the concoction that the patient is getting, it makes your point about 3000 or 30000 remedies quite moot, doesn’t it?”
We don’t know that all water is completely equal in all aspects, all the time, do we? Is it not possible that there may be more dimensions to water than its chemical solutes?
“And you think it is still okay for patients to keep taking this quack nostrum on faith, is that it?”
It’s cheap, it’s virtually risk-free, so why not let them try it? If it works by the placebo mechanism, it still “works,” doesn’t it? If Mr. X’s recurrent gallbladder attacks stop coming without the expense or risk of surgery or drugs for 3 years now, does he care if it is placebo or little nano-elves from Pluto? And if it doesn’t work, there are always the tried and true options to fall back on. So yes, I think it is OK.
Kausik: “Nothing wrong?”
Yep, nothing wrong.
(But why do you make me repeat myself?)
HelenSan:
You haven’t by mistake taken a peek at the link I posted in above in one of my recent responses to you, have you? If not, don’t. Caveat lector! It may do serious damage to your delusions about the ‘risk-free’-ness (I just coined a word!) of homeopathic remedies. And that is but one example.
Ah! This is what a lot of homeopaths, in fact, depend upon – in order to treat their victimspatients. Particularly in countries (such as mine) where ‘alternative medicine’ is assumed to represent radicalism and worn as a badge of honor, especially amongst the rich and the happening, homeopathy is very popular amongst that crowd, because they know that at the first sign of trouble, they would have no problem in being whisked away to a first class hospital (that they can easily afford) and treated with ‘allopathic’ conventional medicine.
But for the misguided poor who are led to believe that homeopathy is a cheaper and better alternative, the picture is not so rosy. In a serious illness, if they don’t get proper treatment with rationality-based medicine, they just die – damn them! – but on the plus side, nobody cares. So homeopathy keeps flourishing.
It is all very fine to look upon rationality-based medicine as a fall-back option, but if treatment in a critical condition (such as in infectious diseases) is delayed, often that significantly raises the morbidity and mortality associated with that condition.
Gaah! I meant this:
victimspatientsWeing: “I was not aware that CAM has not been already explained. More accurate paradigms have made CAM less plausible.”
I mean explanations for CAM that fit with our current paradigms. A future paradigm that emphasizes, say, bioelectromagnetics and/or psychoneuroimmunology in addition to biochemistry, may offer explanations for how homeopathy correlates with desired outcomes, either supplementing current homeopathic philosophy or supplanting it altogether.
Medicine used to believe leeches “worked” because they balanced the humors. Now some in medicine believe leeches may have correlated with desired outcomes because they stimulated circulation. A more accurate paradigm can make an intervention more plausible with a different explanation.
“What implausible anomalies do you have in mind?”
For one, any empirical observation of a significant correlation between CAM and a desired outcome (say in Dr. Brown’s RCT using homeopathic remedy X) might be considered an implausible anomaly.
Kausik: “But for the misguided poor who are led to believe that homeopathy is a cheaper and better alternative, the picture is not so rosy. ”
The problem here is not homeopathy itself, is it? It is misguidance. Make sure people are not misguided, and the problem is solved.
“It is all very fine to look upon rationality-based medicine as a fall-back option, but if treatment in a critical condition (such as in infectious diseases) is delayed, often that significantly raises the morbidity and mortality associated with that condition.”
Again, the problem here is lack of guidance and misjudgment. The time period for “fall-back” is different for different diseases. If my husband has a heart attack, I call 911. Then I might use homeopathy. The “fall back” period is the time it takes for the ambulance to get there. In the time before one receives medical attention, I don’t see the harm in trying CAM.
I am sorry I missed the link under “nothing wrong?” It didn’t look like a link until I searched for it.
In that case, the parents made a bad choice to ignore the observation that homeopathy WAS NOT working. When I say “what’s wrong with that,” I mean what’s wrong with using or trying homeopathy when it is temporally associated with a desired outcome? If the desired outcome isn’t forthcoming, it’s time for the “fall-back”! Failure to take that fall-back, well that isn’t the fault of homeopathy itself–again it is bad judgment on the part of the consumer and provider, if any.
The truth be told, most people turn to homeopathy AFTER fruitless trials with conventional medicine. So the perceived problem that some people won’t fall-back to conventional medicine on time is actually a rare position to begin with. If it weren’t so rare relative to the number of homeopathy consumers, I am sure SBM would be plastered with such anecdotes, not to mention SBM proponents would have gotten homeopathy to be outlawed by now.
“It’s cheap, it’s virtually risk-free, so why not let them try it? If it works by the placebo mechanism, it still ‘works,’ doesn’t it?”
For someone with such a deep concern over the proper attribution of causation in medicine, HelenSan sure seems to loves to fudge what’s causing what (er, I mean what’s “effecting” what, in the practically interpreted but much less rigorous sense of the, um, construct) when it comes to placebo. In the first sentence and before “mechanism” in the second, “it” is a homeopathic remedy. However, to agree that the “it” after “mechanism” still refers to the remedy would require extremely flawed thinking. The “it” that gives the appearance of “working” could be any number of things including coincidence.
HelenSan said
“expecting a higher level of proof from CAM before allowing patients to try it”
Straw man. I don’t remember anyone saying patients should not be allowed to try CAM. We are only saying they should not be told it is more effective than placebo when there is no good scientific evidence that it is. Isn’t that exactly what you’ve been saying about the evidence from clinical trials?
We are not asking for a higher level of proof, we are asking for the same level of proof: treatments in the scientific mainstream are compatible with the rest of scientific knowledge and we only ask the same of CAM treatments.
Harriet Hall: ” I don’t remember anyone saying patients should not be allowed to try CAM.”
You haven’t, to be sure, along with pmoran, weing, and others. It was implied by comments from Kausik Datta, Joe, and A. Noyd. But maybe I misinterpreted their snickering and derision and how using CAM is not “okay.”
In the context of my original post, my plea for “allowing” patients to try CAM was more about recognizing possible effectiveness in event of a positive RCT, as opposed to legal disallowance. You know, “allow” the patient to try homeopathy instead of trying to talk them out of it because the positive RCT can’t possibly be correct.
“We are only saying they should not be told it is more effective than placebo when there is no good scientific evidence that it is. Isn’t that exactly what you’ve been saying about the evidence from clinical trials?”
That *is* exactly what I’ve been saying. And I wholeheartedly agree. Furthermore, I think it would be more accurate in general to speak of effectiveness in terms of correlations and leave the term “scientific” out of it..
“We are not asking for a higher level of proof, we are asking for the same level of proof: treatments in the scientific mainstream are compatible with the rest of scientific knowledge and we only ask the same of CAM treatments.”
The problem is practical “effectiveness,” if you will, can often lag behind the “rest of scientific knowledge.” I am saying, if an RCT shows “effectiveness” beyond placebo for a CAM, using the same design and methodology accepted for conventional medicine, then the results shouldn’t be dismissed outright because the “rest of scientific knowledge” doesn’t currently support it. I think it deserves replication and further investigation. It’s the same point Wondering made at the beginning of this debate.
The bottom line is I don’t think consumers, or even providers, of CAM need to be derided or otherwise attacked because they are trying something other than conventional medicine. I don’t think CAM researchers need to called idiots (or equivalent) because they use the same RCT design conventional medicine uses without the “rest of scientific knowledge” to back them up first.
HelenSan: “The bottom line is I don’t think consumers, or even providers, of CAM need to be derided or otherwise attacked because they are trying something other than conventional medicine. I don’t think CAM researchers need to called idiots (or equivalent) because they use the same RCT design conventional medicine uses without the ‘rest of scientific knowledge’ to back them up first.”
But apparently making up strawman after strawman is just a peachy thing to do, eh? I suppose if you weren’t so adept at missing the point you’d have stabbed yourself to death on it by now.
HelenSan on 05 Oct 2009 at 1:07 pm “… Let’s go back to homeopathy. It is assumed that remedies cannot work because they have no pharmacologically active ingredients. But what if homeopathy works electromagnetically rather than pharmacologically?”
It is equally likely that there are magical elves at work.
You have added “argument from ignorance” to your pontifications. So, to summarize, you are credulous, adept at “straw man” and post modern (New Age) fashionable nonsense arguments, sophomoric, and ignorant. Have I missed anything? Oh, and while you give credence to anecdote, you excoriate scientific research in medicine.
You attribute your “knowledge” of science to being married to a chemist. Well, if he has not disabused you of your delusions, he probably represents the degradation in science after WW2 when the government gave money to academics to do research that taxed the availability of quality students. So, academics began giving doctorates to poor students who would work for a while for low wages.
How could a future paradigm possibly supplant homeopathic philosophy, when it has already been supplanted by science based medicine and shown to be no better than placebo? That’s like saying we should still sacrifice goats to ensure recovery from a pneumonia or have a shamanistic ritual performed so that a plane lands safely. Placebo effects are not implausible. What makes you think the file drawer effect only applies to pharmaceutical studies and not to CAM studies? I think you are trying to justify your adherence to homeopathy by trying to discredit the science behind modern medicine by saying it is not physics or chemistry. Well it is not either of those sciences even though it uses them in practice and is based on them. Forget that physicists will say that chemistry is not a science like they are, etc. Homeopathy uses neither of those sciences. Ask your husband about water memory.
Weing: “What makes you think the file drawer effect only applies to pharmaceutical studies and not to CAM studies?”
I do not think this and never said I did.
” I think you are trying to justify your adherence to homeopathy by trying to discredit the science behind modern medicine by saying it is not physics or chemistry. ”
I justified my use of homeopathy by empirical observations of correlations between use and desired outcome in my own experience. It would not be much different from trying a restaurant, finding satisfaction, and returning to eat there frequently. I do not need to justify what I choose to do in my personal life with science.
So, my discreditation of the “science” behind modern medicine has nothing to do with justifying anything. It has to do with consistency. If you discredit CAM because of poor methodology, then you must discredit conventional medicines that were “proven” with the same methodologies (plausibility notwithstanding). If the current DB-PC-RCT is good enough to “prove effectiveness” with conventional medicine, then it must be good enough to “prove effectiveness” with CAM. Consistency.
“There might be a place for placebos, but good luck finding a way to use them ethically since you have to lie to people to make them work.”
Noyd, it’s not your fault as such, but I’m getting tired of making this point. That is not how statistical effects work. Placebo is not a treatment of anything and has no efficacy in the treatment or prevention of anything, by definition. It is not possible to know in advance how a particular person will react — positively, negatively, or neutrally — to a particular placebo treatment at a particular time in a particular environment. We will only know that their reaction will not be specifically due to the treatment they received, since that treatment did nothing.
I’m starting to call this the M*A*S*H misconception, due to the episode where they run out of morphine and start giving them saline and a winning smile, and it ends up keeping everyone sedated. TV is not reality and that doesn’t work. That’s not the “placebo effect”. There is no such effect which implies that merely convincing people that they are being treated for a non-psychosomatic condition produces any spontaneous efficacy as a result.
This is a persistent social myth that needs to be squished once and for all. There is no placebo effect as the term is commonly understood outside the medical field. That isn’t what placebo effect means. It merely refers to the fact that, by chance alone, a patient’s condition can shift. Even against the principle of regression to the mean. Administering a placebo does not provide “a definite opportunity” for that to happen, and it does, in fact, happen just as often when no placebo has been given.
That placebo is some form of treatment reminds me of nothing so much as the logically mistaken belief that “nothing” is a type of “something”. It’s an inappropriate reification.
HelenSan on 05 Oct 2009 at 6:20 pm “I justified my use of homeopathy by empirical observations of correlations between use and desired outcome in my own experience.”
How presumptuous is that, from someone whose fevered, amateur imagination leads her to “lecture” to scientists?
“If the current DB-PC-RCT is good enough to “prove effectiveness” with conventional medicine, then it must be good enough to “prove effectiveness” with CAM. Consistency.”
True. And it would if it could, but it can’t. CAM is discredited by the same methodology that is applied to SBM. Placebos don’t count. Sorry.
Anecdotes are just that. Check out cognitive biases. Great collection of them on Wikipedia.
Helenscan: The problem is practical “effectiveness,” if you will, can often lag behind the “rest of scientific knowledge.” I am saying, if an RCT shows “effectiveness” beyond placebo for a CAM, using the same design and methodology accepted for conventional medicine, then the results shouldn’t be dismissed outright because the “rest of scientific knowledge” doesn’t currently support it. ”
The “doesn’t currently support it” is a revealing understatement.
The various core principles of homeopathy would never have outlived their pre-scientific origins, if it were not for the placebo and other non-specific benefits that pervade all forms of medical activity and the fact that doing nothing can sometimes work out better than some medical interventions.
If you stand back a bit further from it all, Helenscan, even you may eventually see that homeopathy is part of a very common wider phenomenon, specific to medicine among the sciences, that can ensure the persistence of ideas that lack merit beyond helping to provide the non-specific medical nurture that mankind seems to crave and thrives upon .
I disagree with skeptics who think such medical activity is always a bad thing, but when we have our science hats on we need to be clear-sighted about what is going on. Homeopathy is just one of many such tangents appealing for special consideration.
Weing: “True. And it would if it could, but it can’t. ”
If it can’t, it can’t. I have no problem with that. All I came to this SBM board for is to hear one “True.” And I found one “True.” So thank you. I have no argument with you at this point.
Pmoran: “I disagree with skeptics who think such medical activity is always a bad thing,”
I’m glad to hear that. I’ve read your comments on other threads at SBM advocating some compassion for patients in this regard and have been impressed.
” but when we have our science hats on we need to be clear-sighted about what is going on.”
When I have my science hat on, I say “Medicine is not a science, so one has to interpret the varying correlations in conventional medical and CAM literature in context of the quality of the individual study, as well as the quality of the animal studies, pharmacological studies, and other technological research. At the same time, one has to keep in mind publication bias, the file drawer effect, conflicts of interest between authors/publishers and the pharmaceutical/CAM industry, irrational devotion to dogma, and the possibility of outright fraud.”
HelenSan said,
“I justified my use of homeopathy by empirical observations of correlations between use and desired outcome in my own experience.”
I think the word for this is superstition. The baseball player made the empirical observation that there was a correlation between his lucky underwear and winning the game, so he wore them again.
That is enough to justify trying it again yourself, but of course it can’t say anything about whether it really works. I don’t trust my own empirical observations of correlations and I wonder why you do. That’s the whole reason we have to do science. For centuries doctors trusted their empirical observations of correlations between bloodletting and desired outcomes. And since by your standards, no legitimate evidence shows that it doesn’t work, why not try bloodletting? Are the four humours any less likely than the memory of water? If you go by personal experience and are willing to consider things that science considers unlikely, how can you rationally decide which treatments to try?
Harriet Hall: “That is enough to justify trying it again yourself, but of course it can’t say anything about whether it really works.”
It doesn’t. I completely agree with you on this point.
“And since by your standards, no legitimate evidence shows that it doesn’t work, why not try bloodletting? ”
Because it is also correlated with undesired consequences. There is a cost-benefit debate there. That’s just me.
” If you go by personal experience and are willing to consider things that science considers unlikely, how can you rationally decide which treatments to try?”
I am willing to try anything as long as it is cheap and safe.
If someone hands me a vial of distilled water and says the nano-elves from Pluto swimming inside can heal my disease, and it costs only a dime, I’ll say, what the hell, give me that vial. If he’s selling it for $100, I’ll say no thanks.
Now if I try that 10 cent vial, and it coincides with significant improvement in my disease, I might be willing to pay $10 next time. As soon as it stops “working” (as in being temporally associated with desired outcomes), I’m done buying. Not too hard to figure out.
I don’t have anything against superstition when it doesn’t hurt anyone. In a desperate and difficult world, if someone can find some extra confidence in himself because of his lucky underwear, more power to him. There is a lot of suffering in this world, and sometimes people just need to find comfort where they can. Again, that is just me.
yeahsurewhatever: “Administering a placebo does not provide “a definite opportunity” for that to happen, and it does, in fact, happen just as often when no placebo has been given.”
All the more reason to always include a null control group (a do-nothing control) in every RCT until enough studies show absolutely no differences between the placebo and the null control groups. And if that is true, then we wouldn’t even need placebo controls or double-blind controls anymore. Just use a study group and send the control group home. Think of all the money that would save researchers. Clinical trials would be cheaper. Drugs could get out to patients faster. There is definitely an upside if this were true. I, for one, would be very curious to see the results of such studies.
HelenSan said “I don’t have anything against superstition when it doesn’t hurt anyone.”
Superstition kills. See
http://www.whatstheharm.net/index.html
Helenscan: ” — my discreditation of the “science” behind modern medicine”
Well, all I see an array of fixed misunderstandings.
What takes the cake is your trying to reject medical science through an improper, impossibly absolutist application of the “correlation does not prove causation” card, while not seeming to understand how your own seat-of- the-pants experiences with homeopathy are worm-eaten through and through by each and every one of those “confounders” you have read about somewhere.
Do you not yet understand that clinical trials evolved for this very reason?
Here is YOUR double standard. You wish to ignore the possibility of placebo influences, post hoc vs propter hoc misinterpretations and patient reporting biases when addessing the efficacy of your own personal treatments, yet you wish to clinical sciences to exclude them to a standard that leaves not the least possible room for departure from absolute truth.
This is a nonsense. You must see this. Else, “we will start on our trillionth, trillion-subject, double blinded. placebo-controlled trial when we have raised the first billion dollars”.
Medical science is STILL all about probabilties, no matter how you might wich to avoid applying them to homoepathy.
Pmoran: “What takes the cake is your trying to reject medical science through an improper, impossibly absolutist application of the “correlation does not prove causation” card,”
I am not rejecting it at all! I am advocating a more cautious interpretation that is commensurate with the research design. If you have a correlation, don’t confuse it with causation. I accept all those correlations demonstrated in RCTs. There is no rejection of science going on.
” while not seeming to understand how your own seat-of- the-pants experiences with homeopathy are worm-eaten through and through by each and every one of those “confounders” you have read about somewhere.”
I do understand it and do not dispute it. My personal observations of correlation are just that, correlations (and in a sample size of one, no less) worm eaten with confounders. Have I ever said anything different?
“Here is YOUR double standard. You wish to ignore the possibility of placebo influences, post hoc vs propter hoc misinterpretations and patient reporting biases when addessing the efficacy of your own personal treatments,”
I have not ignored anything. I fully accept the possibility and probability of a host of confounders in my personal observations. I have never remotely addressed the efficacy of my own personal treatments. Please note I have never ever said homeopathy is efficacious, for me or anyone else. I have made it clear that my treatment choice (or restaurant choice) is NOT justified by science and doesn’t need to be.
“yet you wish to clinical sciences to exclude them to a standard that leaves not the least possible room for departure from absolute truth.”"
I have said over and over again that such standards are ethically impossible for clinical research. Since I am not an unethical bastard, I have no wish for clinical research to practice this level of scientific rigor.
All I have ever “wished” is for clinical research to distinguish correlations from causation, to interpret their observational results in context of all their uncontrolled confounders, and advanced claims of “effectiveness” as a correlation and not “truth.”
I don’t want to “reject”or “exclude” the research. I want them to REPORT current research differently. See the difference?
Frankly, it disturbs me a bit that you are inventing all these assertions about my position that I have never said or implied. I expect that from Joe or A. Noyd, but not from you, Pmoran.
Harriet Hall: “Superstition kills.”
And in those cases, I object to the use of superstition in conjunction with terrible judgment and absence of common sense.
Driving kills. Even using a bathtub can kill. But I have no objection to driving or using bathtubs when it doesn’t kill or hurt anyone.
HelenSan on 06 Oct 2009 at 9:35 am “… I want them to REPORT current research differently.”
Yes, everyone is marching out of step except HelenSan, excellent use of capitalization, by the way.
HelenSan on 06 Oct 2009 at 9:42 am “Driving kills.”
One must add “master of the nonsequitur” to your other skills.
HelenSan on 05 Oct 2009 at 1:07 pm
That was a rather intriguing statement, and as Joe pointed out, electromagnetism in this case is as likely a bunch of magical elves. I had dismissed it as a product HelenSan’s fertile imagination.
But holy crap! I found the source of the electromagnetic theory qua homeopathy, following the comment of another poster on another thread! It is based on the work by Luc Montaigner, showing Electromagnetic signals are produced by aqueous nanostructures derived from bacterial DNA (the link to the PDF is here).
Take THAT, you homeopathy disbelievers!!!
I wish I had time now to dissect the jubilant claims of various pro-homeopathy sites, including Avilian (linked above). May be later tonight. As an aside, at 200C dilution of homeopathic drug or DNA, it is still all water (take a look at the paper to see what I mean); in fact, Monteigner et al. acknowledge that at dilutions higher 10^-18 M, the EM signals were indistinguishable from background. But more of that later, hopefully tonight.
I ought to mention that the said poster is Dana Ullman, MPH, who has showed up at various anti-quack websites touting the same study.
The Montaigner group’s paper has nothing to do with homeopathy. If confirmed, it pertains to a very sensitive test for low concentrations of large molecules. The signal is lost before 10^-18 M, which is a roughly 10,000-fold above the Avogadro limit.*
Moreover, in vitro experiments do support homeopathic claims- which must be established in clinical trials.
* The Avogadro number is 6.02 x 10^-23. When you round off, and note that the signal is lost before 10^-18 M, that’s the 10,000.
Helenscan: “All I have ever “wished” is for clinical research to distinguish correlations from causation,”
And you have shown it doesn’t, how?
I am sorry you think I am attacking straw men, for this is an example of one of the “fixed misunderstandings” I referred to.
When not explicit in scientific exchanges, as shown by its usually tentative mode of expression (“the evidence suggests — “) and the provision of gradings for strength of evidence when constructing guidelines for medical practice, the possibility of this and other errors in the interpretation of the evidence is a given.
It is mostly explicit. I urge you to examine the tentative language of the abstracts of mainstream scientific papers.
Weing told you, correctly, that all scientific knowledge is tentative to some degree or other and I have been trying to explain how it is based upon probabilities. All scientists understand this but they may well sometimes take it as read when talking to scientists with the same understanding in their blood or when striving for conciseness on some blog.
Thus ANY claim we make within medicine carries with it an “according to the evidence we have at this time” whether that is stated or not. That covers your obsession with the correlation thing as well as other more common problems posed by the complexities of medical research.
I suggest that, in common with most supporters of CAM, you have always sat some distance away from the scientific action, catching odd snippets of the conversation. It is OK that you may have gotten the wrong impression. It is an incredible level of hubris to conclude that a lot of sincere and intelligent people could have gotten it all wrong.
In case I have to spell this out again, too —
—the probability of treatment efficacy sufficient for the practical purposes of medicine, is established by certain features of study design, statistical analysis, and especially through replicating studies by independent researchers. All three are usually needed.
If the outcome was also predictable from other knowledge of the agent employed then the probabilities are raised even further. The converse applies with entirely implausible treatments which could only “work” as placebo.
Pmoran: “And you have shown it doesn’t, how?”
Read the conclusion of most RCTs. Whenever the conclusion claims that study drug X demonstrated prevention of undesired outcomes or improvement of symptoms, or whenever the conclusion recommends using study drug X to achieve desired outcomes, they are stating a conclusion of causation.
Some authors are more careful, granted. They talk about increased “risk” or how study drug X “may” do Y. I am much more comfortable with those conclusions. Most of them though, do not qualify the limits of their study nearly enough in the discussion.
“I suggest that, in common with most supporters of CAM, you have always sat some distance away from the scientific action, catching odd snippets of the conversation.”
You suggest wrongly. I was trained as a scientist and have had more graduate coursework in research methodology and statistics than most MDs. I read a lot of original papers and am well versed with scientific language. So please spare me your patronization.
Helenscan: “Read the conclusion of most RCTs. Whenever the conclusion claims that study drug X demonstrated prevention of undesired outcomes or improvement of symptoms, or whenever the conclusion recommends using study drug X to achieve desired outcomes, they are stating a conclusion of causation.”
If the authors of a single study don’t allow that they are merely inferring a probability of causation, to a degree that will depend upon the statistical significance achieved and other properties of the research, then they are indeed going too far.
But the blunt assertions you describe are far more common within CAM research than in the mainstream, where it more likely that the conclusions will not go beyond “strongly suggests”. Even intense Cochrane meta-analyses of multiple trials rarely go beyond that.
And that is not quite what I asked. You need to show why with large numbers of subjects, proper randomisation and careful statistical analysis (those things that you dismissively referred to as “elves”, for God’s sake ) a judgment of strong causal probability cannot be arrived at, especially if replicated. That single comment belied your supposed acquaintance with science and statistics.
Pmoran: “You need to show why with large numbers of subjects, proper randomisation and careful statistical analysis (those things that you dismissively referred to as “elves”, for God’s sake ) a judgment of strong causal probability cannot be arrived at, especially if replicated.”
Because causal probability is a lot harder to arrive at than taught in medical school. Because causal probability requires more than large sample sizes and randomization. Because causal probability requires that alternative explanations be eliminated in the study (controlled). Actually eliminated–not just made less likely by large sample sizes and randomization.
Short of that, all you have is correlational probability.
Read Feynman’s article on cargo cult science again. Read the part where he talks about the need to make sure how alternative explanations need to be eliminated. Notice how he didn’t say alternative explanations need to be made less probable by large sample sizes and randomization.
I suggest you look up a physicist or chemist friend or professor you know and trust. And ask them if you can use large sample sizes of say 500 and randomization and statistics instead of active controls to obtain a high level of causal probability. Don’t take my word for it. Ask around–but make sure you ask PhDs who are outside the medical community and were never in it. Get a peek outside the medical culture.
I am reiterating the same argument I’ve made a dozen times already. It’s time to agree to disagree and move on.
” And ask them if you can use large sample sizes of say 500 and randomization and statistics instead of active controls to obtain a high level of causal probability.
I am reiterating the same argument I’ve made a dozen times already ”
But that is not an argument. It is an assertion, and a weird one at that because the use of “active controls” means to me the use of placebos that posses similar side effects to the drug under test as a further refinement of blinding within placebo-controlled trials. It is not a challenge to their basic logic.
Let’s be sure we are on the same page. What precise question do you think we are asking with the usual placebo-controlled trial?
HelenSan on 06 Oct 2009 at 10:22 pm “You suggest wrongly. I was trained as a scientist and have had more graduate coursework in research methodology and statistics than most MDs. I read a lot of original papers and am well versed with scientific language. …”
Wait, wait, wait- what happened to the hubby with the PhD??
No matter. Too many people are “trained” in science and too few are “educated.” Your training is not serving you well, here. The doctors you are engaging here are far better educated than you are.
You may “read” a lot of papers; but you don’t understand them. That’s what Dr. Moran is trying to tell you, more politely than I will.
HelenSan on 07 Oct 2009 at 1:14 am “Read Feynman’s article on cargo cult science again. … I suggest you look up a physicist or chemist friend or professor you know and trust.”
Feynman is a great story teller; but that seems to be the limit of your knowledge.
I am a retired chemistry professor. Trust me- you don’t know what you are typing about. I have had much more sophisticated students in Intro. Chem. than you.
Pmoran: “What precise question do you think we are asking with the usual placebo-controlled trial?”
Does study drug X cause the desired outcome in a significantly higher rate than the placebo drug?
Pmoran: “because the use of “active controls” means to me the use of placebos that posses similar side effects to the drug under test as a further refinement of blinding within placebo-controlled trials.”
I don’t mean pharmacologically active controls. By active controls, I mean the researcher actively does something to control for the confounder, as opposed to letting statistics do the “control” as you propose.
Helenscan, you still haven’t specified what question we are trying to answer.
The clues were there and I should have picked up on the nub of the matter something earlier. I don’t think you understand the difference between OBSERVATIONAL medical studies and INTERVENTIONAL. Your remarks have force with one, but not the other.
You might also ask your chemistry and physics friends and your husband if they understand the difference, if they are sustaining your views. I suspect your collective impression of the norms of medical research derives from the appalling quality of the reporting of correlative observational studies in the media (see Harriet’s recent piece on that), although I cannot pretend that mainstream researchers never respond to certain incentives by puffing up the implications of their research ..
The correlation vs causation error could be called the “Linus Pauling oversight”. Here was a brilliant Nobel-prize winning chemist who came into medical research with little experience but perhaps exaggerated confidence in his general scientific acumen. He concluded from a number of *observational* studies that people with high vitamin C intakes reaped certain health benefits. He was almost single-handedly responsible for the massive over-consumption of vitamins over the last few decades. He could not understand why doctors were skeptical and encouraged sinister explanations of that
We now know, from a number of recently concluded INTERVENTIONAL studies, that he was almost certainly wrong in every one of his claims, and that vitamin intake is merely a marker for a healthier lifestyle in many respects. On present evidence taking multivitamins does not even reproduce the effects of a healthier diet. These studies have weaknesses but NOT because they do not adequately properly control for all variables. Their weakness lies in that they are necessarily relatively short-term, and cannot easily mimic a lifetime of certain behaviours,
Placebo-controlled trials are of necessity interventional. As we have already tried to explained ad nauseam this allows study designs which reasonably control for all possible confounders and, along with other features, permit a very strong inference of causation if results are consistent from study to study. I don’t care if you cannot yet see this, you may do so in time.
I forget how we got onto this from allegations of a double standard in evaluating CAM research, but the answer to that also lies in applying similar probability judgments to research that has an irrefutable theoretical foundation (go for it if, you really think you can show otherwise !!) but which can be easily corrupted. I have explained how we now have some wariness of drug company-funded research.
As a matter of simple fact, no homeopathic approach has yet reached the FDA standard of at least three good quality independent controlled trials showing the same positive results with no negatives. This holds despite the reasonable presumption that researchers into homeopathy are choosing to test out their most certain beliefs. Many of the negative studies involve individualized remedies within homeopathy..
Pmoran: “Helenscan, you still haven’t specified what question we are trying to answer.”
Does study drug X cause the desired outcome in a significantly higher rate than the placebo drug?
Also, very minor point: the name is Helen San, and not Helen scan.
” I don’t think you understand the difference between OBSERVATIONAL medical studies and INTERVENTIONAL.”
RCT’s are interventional. The study that found correlations between autism and television watching are observational. See? I do understand.
” Your remarks have force with one, but not the other.”
My remarks apply to both. But I was focusing on interventional studies like RCTs.
“He concluded from a number of *observational* studies that people with high vitamin C intakes reaped certain health benefits.”
His original study was observational, but he followed it up with a series of interventional RCTs. For example, see here.
http://www.pnas.org/content/75/9/4538.abstract
Helen-San, I have said all I have to say, and my opinion that you do not understand the enormous differences between observational and interventional studies still stands.
Pmoran: “Helenscan, you still haven’t specified what question we are trying to answer.”
I am curious what you think the correct answer is.
Pmoran: “my opinion that you do not understand the enormous differences between observational and interventional studies still stands.”
Okay! Let me put my brain away and just take your word for it.
I’ve explained what I think are the differences between observational and interventional. Why don’t you explain why you think I was wrong?
Helenscan: “I am curious what you think the correct answer is.”
There was no correct answer. I just wanted to know if you thought your comments applied with controlled trials, as opposed to uncontrolled or observational studies or whether you had some pedantic interpretation of what it means to say that a treatment “works better than placebo”.
No, you seem to merely think that “controlled” in Randomised controled trial means “uncontrolled”.
Helen-San” I’ve explained what I think are the differences between observational and interventional. Why don’t you explain why you think I was wrong?”
I didn’t say you were wrong. It is not at all clear that you understand the implications.
Observe how randomization to the use of vitamins prevented that being merely reflecting a choice of the more health-conscious, who will then have better health outcomes for this separate reason.
Proper randomization will prevents a treatment appearing to be effective because it happened to be used by those with more minor illnesses.
There are innumerable possible reasons why retrospective and observational studies may throw up spurious associations and we cannot expect to know them all in advance.
.
Pmoran: “No, you seem to merely think that “controlled” in Randomised controled trial means “uncontrolled”.”
Control is not an on/off switch, either “controlled” or “uncontrolled.” Control is a gradient, with few controls on one end and total control on the other.
Most medical RCTs have controls to be sure, but only a few of them. They are on the control-”lite” end of the spectrum. It really wouldn’t be that much more trouble to add a few more controls (like intermittent drug testing, or a questionnaire about use of other therapies during the study period), but most medical researchers seem to think a few controls are sufficient, and the–excuse me–”Confounder Elves” can take care of the rest.
“Observe how randomization to the use of vitamins prevented that being merely reflecting a choice of the more health-conscious, who will then have better health outcomes for this separate reason.”
I have addressed this issue in previous posts to Weing (I guess you really don’t read my posts to other people!). I discussed at length how randomization prevents selection bias. So you have two groups who start out as equal cohorts. But you have no guarantee that they will STAY equal throughout the study period. Maybe no one at the outset are heroin users, so you think heroin use is controlled for by randomization; but without a true control, you have no way of knowing whether some became heroin users DURING the study. Randomization cannot control for individual choices that appear in the middle of a study.
HelenSan,
I have not yet all the comments in this thread, so I don’t know if this has been pointed out yet, but you are using ‘confounder’ incorrectly.
A confounder is not simply an independent variable, or even more specifically an unmeasured independent variable. It is a variable which is associated with both the dependent variable (the outcome measure) and the independent variable under consideration, but not as an intermediate in a causal chain. It can lead to a spurious appearance of an association between the independent variable and the dependent variable. So “socioeconomic status, marital status, family support, stressors, health insurance, etc.” are not confounders simply because they may influence the outcome.
You are mixing up the idea of controlled and uncontrolled with measured and unmeasured, as well as mixing up the difference between holding parameters equal vs. holding variation equal.
Also, the study you linked to from Linus Pauling was not an RCT. It was actually an observational study.
http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=336151&blobtype=pdf
Linda
“Also, the study you linked to from Linus Pauling was not an RCT. It was actually an observational study.”
Thanks for the comment, and that is true, Linda. That study did not involve randomization of patients, it relied upon subjective and potentially inconsistent definitions of what represents “terminal” cancer, and all data including the selection of controls was obtained rettrospectively. None of these apply in the usual RCT.
The results have also now been contradicted by a number of proper RCTs.
Incidentally, Helen-San, when the first of these other studies came out, Pauling did not dispute the validity of the RCT format. He alleged that Vitamin C did not work because some of the patients had also received chemotherapy. So a second trial was perfomed on patients not receiving chemotherapy with the same outcome. Since then other studies have confirmed the lack of obvious beneifts in cancer from vitamins of any kind and we now know oral vitamin C is poorly absorbed beyond about 1G per day.
fls: “A confounder is not simply an independent variable, or even more specifically an unmeasured independent variable. ”
I never said a confounder is an independent variable. In any study, there is only ONE independent variable, and that is the variable you’re manipulating. There are dependent variables, and those are the variables you are measuring. The other variables you are NOT manipulating, but that influence the dependent variables–those are confounders. So I agree with you, but the correction is misplaced.
“It is a variable which is associated with both the dependent variable (the outcome measure) and the independent variable under consideration, ”
That’s right…
“but not as an intermediate in a causal chain. ”
Do you mean that if sexual intercourse plus heterosexual coition plus fertility plus conception plus implantation plus gestation plus delivery forms the causal “chain” for babies, then fertility is not a confounder in a study of the effect of sexual intercourse on the number of births?
Now imagine you did not know this causal chain or the mechanism involved, and you wanted to study if the number of times one has sexual intercourse has an effect on the number of babies that resulted 9 months later. You randomize 200 couples. You tell one group to have sex everyday of the week. You tell the other group to have sex only once a week. You don’t control for heterosexuality, fertility, prenatal care, etc. You come back 9 months later and find that the weekly group actually had 2x the number of babies as the daily group. Can you conclude that having sex less often causes the appearance of babies? No, because all those uncontrolled variables in the causal chain confounded the study. The weekly group may have very well been biased in any of those causal chain variables, which then influenced the observed result.
“It can lead to a spurious appearance of an association between the independent variable and the dependent variable.”
This is only one kind of confounder. Confounders exist in experimental designs as well. See the example above.
” So “socioeconomic status, marital status, family support, stressors, health insurance, etc.” are not confounders simply because they may influence the outcome.”
Would you say that a variable that may influence the outcome, such as stressors on health, would be correlated with the outcome, along with the independent variable? Can you see how the 3 correlations can lead to a spurious association between the independent and dependent variables?
“Also, the study you linked to from Linus Pauling was not an RCT. It was actually an observational study.”
It was a controlled trial, which is not an observational study but an interventional one. My error was overlooking the fact that it was not randomized. Thank you for that correction. But my point still stands. Even though it wasn’t randomized, it was still an interventional study.
HelenSan on 08 Oct 2009 at 6:26 pm “It was a controlled trial, which is not an observational study but an interventional one.”
The study you cited:
http://www.pnas.org/content/75/9/4538.abstract
Is a pilot study which is not placebo controlled, nor blinded. Moreover, PNAS is not a refereed journal since members of the NAS (Pauling) can simply have their ideas published.
A scientist (such as yourself) addressing this issue should point to the best support. If, after 30+ years, there is no better support than that meager study, their conclusions are likely erroneous. So, can you find a definitive study, or, even, a good one?
HelenSan,
All the variables you describe are independent variables. They are the variables upon which the outcome depends, which influence the outcome. The variable which you specifically manipulate is one of the independent variables, and it may be the only variable used for your analysis. And it is worthwhile to distinguish it from other variables by name, by calling it the “experimental variable”, for example. But “independent” refers to a class. Other names for variables in this class are “covariates”, “risk factors”, “controls”, “regressors”, “explanatory variables”, etc. The variables that you measure (age, sex, socio-economic class, etc.) are also independent variables because the effect of using them in multiple regression or using them as controls is the same as the effect of manipulation.
A confounder is simply not one of the members of this class, but is a variable which has an association (not necessarily causal) between both the outcome and the purported independent variable under consideration, but definitely not as an intermediate in a causal chain. For example, the relationship between drinking coffee (independent variable) and lung cancer (dependent variable) is confounded by cigarette smoking. Coffee-drinking and smoking may vary together within a population, so that measures of lung cancer within that population will show that it varies with coffee-drinking, even though coffee-drinking does not cause smoking, or smoking cause coffee-drinking, and coffee-drinking does not cause lung cancer.
Variables which simply influence the outcome are not what is meant by the term “confounder”, and it is your use of the term to mean that that is incorrect.
Getting to your specific example:
HelenSan:
“Do you mean that if sexual intercourse plus heterosexual coition plus fertility plus conception plus implantation plus gestation plus delivery forms the causal “chain” for babies, then fertility is not a confounder in a study of the effect of sexual intercourse on the number of births?”
Unless you think that sexual intercourse varies with fertility, such that sexual intercourse does not have the effect of producing babies it only has the appearance of producing babies because it is associated (in this population) with fertility, then “no”, fertility is not a confounder in your example.
HelenSan:
“You come back 9 months later and find that the weekly group actually had 2x the number of babies as the daily group. Can you conclude that having sex less often causes the appearance of babies? No, because all those uncontrolled variables in the causal chain confounded the study. The weekly group may have very well been biased in any of those causal chain variables, which then influenced the observed result.”
This is an example where you have confused the idea of unmeasured variables with the idea of uncontrolled variables. And you have confused the idea of unmeasured variables with confounders. What you are describing is not what is meant when people (who know what they are talking about) use the word “confounders”.
Now don’t get me wrong. The issue of other variables which influence the results is relevant whenever we are asking these questions. But using technical terms incorrectly means that you will get busted if you try to claim any sort of expertise.
HelenSan:
“It was a controlled trial, which is not an observational study but an interventional one. My error was overlooking the fact that it was not randomized. Thank you for that correction. But my point still stands. Even though it wasn’t randomized, it was still an interventional study.”
The description is very clearly that of what is called a cohort study (a type of observational study). I suspect that what mixes you up is that the exposure in question is of a type that is used as an intervention in some interventional studies. But “interventional” is not determined by whether or not what a group was exposed to was an intervention. Instead, an interventional study collects a group for the purpose of intervening in the usual course of events. Now the choice of how that intervention is to be applied may be randomized or not, but this is not how the determination is made as to whether it’s an interventional vs. observational trial.
The simplest way to come to this realization for this particular study is to recognize that the groups were collected after (and on the basis of that exposure) they had been exposed to vitamin C (or not), not before. This case is pretty clear cut, which means that you have just confirmed pmoran’s suspicions.
Pmoran: “That study did not involve randomization of patients, it relied upon subjective and potentially inconsistent definitions of what represents “terminal” cancer, and all data including the selection of controls was obtained rettrospectively. None of these apply in the usual RCT.”
I don’t dispute that this study was poorly executed and poorly written–exactly the kind of crap pseudoscience I’ve been criticizing throughout this thread. It is sort of sad that someone with the scientific acumen of Pauling would go completely woo when it came to human subjects, but that’s really beside the point of our original debate. You brought it up as an example of an observational study. I said it was interventional. Redirect.
Observational studies are where you observe correlations as they naturally occur in the population. You look, but don’t touch, so to speak. In scientific speak, they are usually called correlational studies.
Interventional studies are where you manipulate the independent variable. You decide who who gets it and who doesn’t. In scientific speak, interventional studies are usually called experiments.
Correlational studies are interpreted very carefully and no interpretation of causation is ever drawn because of lack of control of confounders. In experiments, confounders are usually controlled such that researchers do interpret causation. Experiments that are confounded are usually thrown out as uninterpretable.
I argued that medical RCTs are so poorly controlled that these experiments are all confounded, and causation cannot be interpreted. Now instead of throwing them out as completely useless, I said one could interpret the results very carefully AS IF they were correlational studies, with the same level of confidence as a correlational study. I qualified this at the beginning of the thread.
Towards the end, I just called the confounded RCTs “correlational studies.” What I still meant was that they are as confounded as a correlational study and should be interpreted as one. My point was a poorly controlled experiment might as well be a correlational study.
This may be the source of the confusion. If you had forgotten my earlier posts and read my later posts out of that context, it may appear that I don’t know the difference between a correlational study and an experiment.
Joe, you said:
“The study you cited:
http://www.pnas.org/content/75/9/4538.abstract
Is a pilot study which is not placebo controlled, nor blinded.”
At issue is not whether or not it was placebo controlled or blinded (parts of it were), but whether the groups were collected for the intent of intervening, or whether they were collected on the basis of exposure to a prior intervention. It was clearly the latter. The abstract even states this, let alone the body of the paper (http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=336151&blobtype=pdf). I wouldn’t want you to leave the impression that this ‘doesn’t count’ because the quality is poor. It wouldn’t count even if the quality were outstanding.
Linda
HelenSan, you wrote:
“Now imagine you did not know this causal chain or the mechanism involved, and you wanted to study if the number of times one has sexual intercourse has an effect on the number of babies that resulted 9 months later. You randomize 200 couples. You tell one group to have sex everyday of the week. You tell the other group to have sex only once a week. You don’t control for heterosexuality, fertility, prenatal care, etc. You come back 9 months later and find that the weekly group actually had 2x the number of babies as the daily group. Can you conclude that having sex less often causes the appearance of babies? No, because all those uncontrolled variables in the causal chain confounded the study. The weekly group may have very well been biased in any of those causal chain variables, which then influenced the observed result.”
The unmeasured variables in the causal chain could be confounders if any of them were associated with the frequency of sexual intercourse. That’s the difference between a variable which influences the outcome and a variable which confounds the outcome.
fls: “All the variables you describe are independent variables. They are the variables upon which the outcome depends, which influence the outcome. ”
I beg to differ. In experiments, independent variables are the variables you manipulate. Other variables are called extraneous variables. Some extraneous variables have nothing to do with anything. Some extraneous variables influence the dependent variable. Those that influence the dependent variable, but are NOT the independent variable you are manipulating, are called “confounders.”
http://en.wikipedia.org/wiki/Dependent_and_independent_variables
http://en.wikipedia.org/wiki/Confounder
I realize wikipedia is not the final authority on the definition of words, but it’ll give you an idea of what I’m talking about.
“Variables which simply influence the outcome are not what is meant by the term “confounder”, and it is your use of the term to mean that that is incorrect.”
Well, we can get into a “Your definition is wrong. No, YOUR definition is wrong” debate. But it won’t be resolved without a mediating authority that we both respect. Plus I don’t see the point.
“This is an example where you have confused the idea of unmeasured variables with the idea of uncontrolled variables. And you have confused the idea of unmeasured variables with confounders. ”
Measurement is not the same as control. And unmeasured is not the same as confounding. I agree.
In the intercourse frequency experiment, if you were to control for fertility, you would have to measure it first. Then in addition to measurement, you would have to make sure all couples were fertile, or at the very least, that the same ratio of fertile to infertile couples existed in both the study and control groups (so as to eliminate bias).
Now if you are not measuring for fertility, you are definitely not controlling for it. And the results of the experiment are confounded. You cannot attribute causation in an experiment where alternative explanations (confounders) were not eliminated. Definition bickering aside, THAT point still remains.
“What you are describing is not what is meant when people (who know what they are talking about) use the word “confounders”.”
Perhaps you can supply a reference to back this up? Also to demonstrate who these “people” are.
“But using technical terms incorrectly means that you will get busted if you try to claim any sort of expertise.”
Well, I’d say you’re busted then.
Listen, you can call me an ignorant hack as articulately as you want. And I can even return the favor, though I really try not to go there. But I fail to see how such personal attacks further the debate.
Argue with logic and reason. Short of that, argue with references. Telling me I don’t know what I’m talking about–well, that’s just your opinion. Saying it over and over again doesn’t make it true.
“The simplest way to come to this realization for this particular study is to recognize that the groups were collected after (and on the basis of that exposure) they had been exposed to vitamin C (or not), not before. ”
If this group was indeed collected after exposure, and the independent variable was not manipulated, then I would agree with you. Retrospective studies are observational and correlational.
The confusion arises because Pauling got involved in this research retrospectively. However, the original researchers (Cameron and others) and providers chose who received ascorbate and who didn’t. The independent variable was manipulated, and the outcome was measured *prospectively.” Pauling then reevaluated all this data along with selection of controls retrospectively and made a big mish mash of things. But the trial of the study group was still prospective.
“The treated group consists of 100 patients who began ascorbate treatment, as described by Cameron and Campbell (4) (usually 10 g/day, by intravenous infusion for about 10 days and orally thereafter), at the time in the progress of their disease when in the considered opinion of at least two independent clinicians the continuance of any conventional form of treatment would offer no further benefit.
Fifty of the treated subjects are those described in ref. 4 and the other 50 were obtained by random selection from the alphabetical index of ascorbate-treated patients in Vale of Leven District General Hospital, where treatment of some terminal cancer patients with ascorbate has been under clinical trial since November 1971. We believe that the ascorbate-treated patients represent a random selection of all of the terminal patients in this hospital, even though no formal randomization process wasused….
Treatment of terminal cancer patients with ascorbate was
cautiously begun in November 1971, for reasons discussed in
our earlier papers (1, 2), and has been continued because it
seemed to have some value (4, 5). Once the practice had become locally established, the selection of a patient for treatment with ascorbate was often initiated by one of the younger surgeons (the Registrars), as they became familiar with the idea and convinced of its worth.”
The important point is the patients did not choose to use ascorbate themselves. Now there could be a provider bias in who they chose to receive ascorbate, and that definitely confounds the results. But the study is still interventional. That is why it was called a clinical “trial” and not merely an observation of naturally occurring correlations.
Truth be told, it is a big mess and a very poor example of the difference between observational and interventional.
“I argued that medical RCTs are so poorly controlled that these experiments are all confounded, and causation cannot be interpreted. Now instead of throwing them out as completely useless, I said one could interpret the results very carefully AS IF they were correlational studies, with the same level of confidence as a correlational study. I qualified this at the beginning of the thread.”
Not really. You simply made a claim. You really didn’t support the claim, unless I missed it, with an example of an RCT. That would technically not be an argument.
fls: “The unmeasured variables in the causal chain could be confounders if any of them were associated with the frequency of sexual intercourse. That’s the difference between a variable which influences the outcome and a variable which confounds the outcome.”
Technically, that is correct. A confounder has to be correlated with BOTH the independent and dependent variable, and not just the dependent variable.
However, without studies showing there is NO correlation between fertility and the frequency of sex, you cannot assume that fertility is NOT a confounder. Without proof that it is NOT a confounder, you need to control for it as if it could be.
If you are running an RCT, and you don’t want to control for use of recreational drugs, you need to prove it is not a confounder by citing at least one study that show there is no correlation between your study drug X and recreational drugs. Of course, no such studies exist with most study drugs, so you have to consider it a POSSIBLE confounder and control for it.
Yes, I could have been more technically precise by qualifying “confounders” as “possible confounders.” But it still doesn’t change the thrust of my arguments.
Weing: “You really didn’t support the claim, unless I missed it, with an example of an RCT. ”
I supported the claim with hypothetical examples of Dr. Brown’s RCT, Dr. Smith’s RCT, etc.
If you want me to support the claim with a real RCT, give me a full text of a study and I’ll tell you why it was poorly controlled and why interpretations of causation are not warranted or supported by the data. I’ve made this offer before.
HelenSan,
I’m not talking about a term whose definition is vague or under dispute. “Confounder” has a specific technical meaning, regardless of whether or not you were ignorant of that technical meaning. I would like to point out that the wikipedia article does confirm that meaning, in contradiction to your claim of what it means.
You are simply describing different types of independent variables. I will agree that you can find sources, like wikipedia, which like to differentiate between one kind of variable – that which is deliberately introduced by the experimenter – and other kinds of variables – those whose introduction is not under the direct control of the experimenter – and to call the former “independent”. And it is reasonable to make that distinction, using whatever label helps to clarify what it is you are referring to. The point is that independent variables are all of a sort whose manipulation (whether that manipulation is through the introduction of the variable or through measurement or control) influences the dependent variable. It’s okay if you want to call it something else, I’m really just interested in conveying the concept. The concept that I refer to as “independent variables” and which you call “extraneous variables” (your wikipedia link refers to this as a synonym of independent variables) is different from the concept of confounding variables.
I’m not attacking you. I am pointing out that you are using a term differently from its technical use. And when you write articles that are supposed to make it look like you have some sort of technical expertise, this misuse will either misinform people who so not know any better, or will serve to reveal a degree of ignorance on your part to those who do know better. Pretending that this is merely a matter of opinion only serves to highlight the problem.
HelanSan:
“The confusion arises because Pauling got involved in this research retrospectively. However, the original researchers (Cameron and others) and providers chose who received ascorbate and who didn’t. The independent variable was manipulated, and the outcome was measured *prospectively.” Pauling then reevaluated all this data along with selection of controls retrospectively and made a big mish mash of things. But the trial of the study group was still prospective.”
Yes, it was my impression that the description of the original instigation of the treatment was what gave you the idea that this was an interventional study. But this simply represents the formation of a cohort with a particular exposure. If the group had been formed on the basis of some other variable (in this case, it could be “untreatability”) and then an intervention had been assigned, then the intervention could be considered a variable. However, the group was formed on the basis of the exposure, i.e. “treated with ascorbate”, which means that the intervention was not a variable. They were given ascorbate if they happened to be under Dr. Cameron’s care or were referred to his care and they were untreatable. But they were not deliberately chosen from a broader cohort of “untreatable” patients with the intervention applied to some, but not others. All comers were treated (as described in earlier papers).
HelenSan:
“The important point is the patients did not choose to use ascorbate themselves. Now there could be a provider bias in who they chose to receive ascorbate, and that definitely confounds the results. But the study is still interventional. That is why it was called a clinical “trial” and not merely an observation of naturally occurring correlations.”
Whether or not the patients chose to use ascorbate themselves is not relevant. At issue is whether the use of ascorbate varied, and it did not. All persons in the cohort received ascorbate. Of note, Cameron calls it “uncontrolled clinical observations” in his original description, which is accurate. However, whether or not Cameron or Pauling choose to call it a clinical trial is not the determining characteristic of whether it is interventional.
http://www.cellmedsoc.org/research_archive/NHC/studien_pdf/old/the_orthomolecular_treatment_of_cancer.pdf
“I supported the claim with hypothetical examples of Dr. Brown’s RCT, Dr. Smith’s RCT, etc.”
That’s not really supporting a claim. Do you mean you haven’t reviewed any real RCTs? Get a recent issue of NEJM and check the methodology section to see if they were RCTs and review them.
If you think that we are practicing cargo cult medicine, then drink a bottle of citrate of magnesia and see if has an effect or not. Don’t drink a bottle of water that is just labeled citrate of magnesia, that would be placebo, although the taste would be a giveaway.
HelenSan:
“Technically, that is correct. A confounder has to be correlated with BOTH the independent and dependent variable, and not just the dependent variable.
However, without studies showing there is NO correlation between fertility and the frequency of sex, you cannot assume that fertility is NOT a confounder. Without proof that it is NOT a confounder, you need to control for it as if it could be.
If you are running an RCT, and you don’t want to control for use of recreational drugs, you need to prove it is not a confounder by citing at least one study that show there is no correlation between your study drug X and recreational drugs. Of course, no such studies exist with most study drugs, so you have to consider it a POSSIBLE confounder and control for it.”
Wow.
No. You can’t have an association between a RANDOMLY-ASSIGNED variable and a confounder.
Think about it. If there is an association between sex-frequency and fertility, then sex-frequency, because it was assigned randomly, must influence fertility, which makes fertility part of a causal chain, not a confounder. If the distribution of fertility was unequal between the groups, then fertility is associated with the randomization process, not with the intervention and therefore is not a confounder.
If more people taking drug X use recreational drugs, they either started out that way (a randomization issue, not a confounder) or drug X influenced recreational drug use (not a confounder).
We don’t need studies showing no association, because IT CANNOT BE A CONFOUNDER IF THE INTERVENTION IS RANDOMLY ASSIGNED.
If the Pauling/Cameron study is to be considered an interventional study, then the intervention under consideration is “coming to the attention of Dr. Cameron or his students”, not “ascorbic acid”.
““Confounder” has a specific technical meaning, regardless of whether or not you were ignorant of that technical meaning.
I would like to point out that the wikipedia article does confirm that meaning, in contradiction to your claim of what it means.”
And I have no dispute with the meaning described in wikipedia. Yes, “confounder” has a technical meaning, but saying I am ignorant of it doesn’t make it so. I have conceded that I should have been more precise and said “possible confounder” instead of plain old “confounder.” With that in mind, prove to me where I used “possible confounder” inaccurately.
“The concept that I refer to as “independent variables” and which you call “extraneous variables” (your wikipedia link refers to this as a synonym of independent variables) is different from the concept of confounding variables.”
Well, ok, if you want to say extraneous variables are statistically independent, I see where you’re coming from. In the scientific communities that I have participated in, the usage of “independent” variable has always been that of the experimental variable. So, whatever. But if extraneous variables are correlated with both the independent variable (the one under manipulation) and the dependent variable, then they are no longer statistically independent and are called “confounders.” If those extraneous variables COULD BE correlated with both, then they are POSSIBLE confounders. That is why even though extraneous variables could be statistically independent, no practicing researcher I know calls them “independent variables” because there is rarely any PROOF that they are not correlated with both the IV and the DV. One cannot assume that they are independent–so it is just safer to call them extraneous.
” I am pointing out that you are using a term differently from its technical use. ”
But I am not. How I use it is exactly how practicing scientists use those terms. Again, you want to accuse me of ignorance, you should copy and paste where I used it differently from its technical definition.
“Pretending that this is merely a matter of opinion only serves to highlight the problem.”
Wow. Your resort to strawmen knows no bounds, does it?
When did I ever pretend the definitions were a matter of opinion? I said your assertions of the “true” definitions were your opinion. as you cited no references. I said your insinuation that I am ignorant is a matter of opinion.
Now I cited the wikipedia articles as definitions that I don’t have problems with. So we have some objective standard by which to measure whether my statements are inconsistent with them or not. You want to accuse me of something, the burden of proof of my ignorance is on you. It is not as easy as coming in and making a pronouncement.
I really don’t understand this method of debate. I make an assertion and explain it. You guys (not just you Linda, but everyone here has done it) tell me I believe something else that I didn’t say, and then accuse me of ignorance or worse. I try again to explain and clarify my position. But it doesn’t matter. It’s like a broken-record strawman reiterated with deliberate obtuseness. There is no integrity in this type of debate.
Helen-San: ?I argued that medical RCTs are so poorly controlled that these experiments are all confounded, and causation cannot be interpreted. Now instead of throwing them out as completely useless, I said one could interpret the results very carefully AS IF they were correlational studies, with the same level of confidence as a correlational study. I qualified this at the beginning of the thread. ”
“– same level of confidence –”? So randomisation of even very large numbers of subjects to an intervention does not in any way increase the likelihood that a statistically significant difference in outcomes (say p<.007) is being induced by that intervention?
I doubt if you will find anyone who will agree with you. There is something you are not quite getting.
“Think about it. If there is an association between sex-frequency and fertility, then sex-frequency, because it was assigned randomly, must influence fertility, which makes fertility part of a causal chain, not a confounder. ”
If fertility is part of the causal chain, then it is correlated with BOTH the independent and dependent variables, which makes it by definition a confounder.
“If the distribution of fertility was unequal between the groups, then fertility is associated with the randomization process, not with the intervention and therefore is not a confounder.”
As I have said before, randomization attempts to eliminate selection bias. I say attempt because bias can still slip through. In addition, any variable that can change AFTER selection is not “controlled” by randomization. Now variables can bias the study or control groups AND be correlated with the IV and DV at the same time (and be a confounder).
This debate about definitions and usage cannot advance unless we agree on a research methodology and/or statistics text as arbiter. Otherwise, all we have are conflicting assertions that aren’t true just because we say they are.
Frankly, I am tired of it. It has been a long thread, and at this point, we’ve all made the same points several different ways. The accusations that misconstrue my positions apparently aren’t going to stop. No one here is ever going to agree with me about confounders in an RCT. I see no point in continuing this conversation. My husband is complaining that I am wasting my time and need to call it quits. So I think finally, I am going to listen to him.
Thank you Weing and Wondering, anyway, for being the only ones to give me hope that some of you guys can be objective about CAM.
Pmoran: “I doubt if you will find anyone who will agree with you. ”
Plenty of scientists I know agree with me. There’s just none here on SBM.
I would like to thank you and everyone here for taking the time to debate with me and help me understand your perspective. For the most part, it’s been a pleasure for me–even if you can’t say the same.
“Plenty of scientists I know agree with me.”
Produce one. I think you have misunderstood what they are saying, too,
“As I have said before, randomization attempts to eliminate selection bias. I say attempt because bias can still slip through. In addition, any variable that can change AFTER selection is not “controlled” by randomization. Now variables can bias the study or control groups AND be correlated with the IV and DV at the same time (and be a confounder).”
If you are departing, I must point out before you go that you are now shifting ground.
You are now referring to matters that any quality RCT guards against at the design stage and checks upon routinely.
This is no challenge to the core principles on which RCTs are based, You have completely rejected those without being able to offer clear explanation or explicit examples. Don’t pretend that you have.
And do put your scientist husband on if he is prepared to support your position. Most of us have been very polite with you, even though your comments suggest that we, along with everyone involved in medical research, haven’t a clue about it.
HelenSan on 09 Oct 2009 at 7:46 pm “Plenty of scientists I know agree with me. There’s just none here on SBM.”
That may well be true. It is common for scientists in one field to doubt the diligence of scientists in another field because they don’t know the background and practices.
HelenSan:
“And I have no dispute with the meaning described in wikipedia. Yes, “confounder” has a technical meaning, but saying I am ignorant of it doesn’t make it so. I have conceded that I should have been more precise and said “possible confounder” instead of plain old “confounder.” With that in mind, prove to me where I used “possible confounder” inaccurately.”
Where you said in a subsequent post:
“If fertility is part of the causal chain, then it is correlated with BOTH the independent and dependent variables, which makes it by definition a confounder.”
From wikipedia:
“The confounder is not allowed to lie in the causal pathway…”
You still do not seem to understand how the term is used. And this is not a trivial issue, because your entire argument rests on the idea that if a randomly-assigned intervention were confounded, our failure to discover this would leave us vulnerable. But this skips over the fundamental problem that it is not possible to confound a randomly-assigned intervention in the manner you describe.
HelenSan:
“As I have said before, randomization attempts to eliminate selection bias. I say attempt because bias can still slip through. In addition, any variable that can change AFTER selection is not “controlled” by randomization. Now variables can bias the study or control groups AND be correlated with the IV and DV at the same time (and be a confounder).”
The intervention is applied RANDOMLY. Regardless of whether groups characteristics are distributed evenly or unevenly by the randomization process, a RANDOM variable cannot show an association with any other variable except by CHANCE. And a CHANCE association cannot be a confounder, because the association will disappear when one goes to “prove it is not a confounder”.
Also, the randomization process controls variables both before and after the process of the assignment of the intervention. You are confusing the control of parameters with the control of variation.
As to your complaints about the form which this discussion has taken…how could it be otherwise? If you say things that are wrong, they will be corrected. If you are missing key information, it will be provided. It makes no sense to complain when you are shown to be wrong, or to complain when you missed out on some key information, if your argument is based on that wrong or missing information.
fls: ““The confounder is not allowed to lie in the causal pathway…”
The rest of the quotation says: “If A is thought to be the cause of disease C, the confounding variable B may not be SOLELY caused by behaviour A; and behaviour B shall not ALWAYS lead to behaviour C.” (Emphasis mine.)
So unless sex-frequency is the *only* cause of fertility, and fertility *always* leads to babies, fertility is not excluded as a confounder.
In other words, “causal pathway,” as an exclusionary criterion for the use of the term “confounder,” is very narrowly defined.
I previously said: “If fertility is part of the causal chain, then it is correlated with BOTH the independent and dependent variables, which makes it by definition a confounder.”
There’s causal chain, and then there’s *causal chain*. I am pointing out that something can be *one* of the causal factors, thus correlated with both the IV and DV, without being the *kind* of causal factor that is excluded as a “confounder.” I am sorry I wasn’t more precise and thorough in my explanation.
“You still do not seem to understand how the term is used.”
Ah, but I do. I just don’t use the term in the same way you do. Let’s not confuse absence of agreement with absence of understanding again.
To be fair, I understand your point of view. I researched some medical lectures yesterday and found exactly your arguments. What you are telling me is what medical school teaches as correct and true. I get that. What they don’t teach are the finer conditions under which such things are correct and true. So the students learn a sort of absolute rule, and here I am, appearing to break that rule.
What I hope you or anyone on this board can “get” is that I was taught differently in graduate school, and so was my husband and other scientists we know. What we have are different “scientific” cultures, each with its own “rules” of what the terms mean, how to use statistics, and what constitutes scientific rigor. Who’s right? Well, I think my way is right and medicine is wrong to use so few controls, and you think I’m ignorant. So there we are.
“But this skips over the fundamental problem that it is not possible to confound a randomly-assigned intervention in the manner you describe.
You say it is not possible for variables such as recreational drug use or socioeconomic status (SES) to be confounders because the technical use of the term “confounder” requires that (1) they would have to be correlated with both the IV and DV, and (2) they cannot be part of the causal chain.
(1) Imagine an RCT with poor and rich subjects assigned in equal proportions at the outset of the study. A bunch of subjects drop out of the study in the middle, and it turns out most of them were rich and most of them were in the placebo group. When they did, an SES bias occurred in the study that wasn’t there at the beginning. The placebo group is now comprised a higher ratio of poor subjects, and it turns out they had poorer diets and less healthy lifestyles than the study group. High SES, in this situation, becomes positively correlated with both the study group and better outcome. The DV could be reflecting the SES bias rather than the true effects of the study drug. Therefore, SES is a possible confounder.
What I keep saying is while randomization is indeed very useful, it cannot control biases that can occur DURING a study. So we need extra controls in addition to randomization.
(2) Unless the study drug is the sole cause of SES or recreational drug use, AND SES status or recreational drug use always leads to the measured outcome, neither SES nor recreational drug use are excluded as “confounders.”
OK, I’m gone. Really. You guys have fun now.
HelenSan:
“To be fair, I understand your point of view. I researched some medical lectures yesterday and found exactly your arguments. What you are telling me is what medical school teaches as correct and true. I get that. What they don’t teach are the finer conditions under which such things are correct and true. So the students learn a sort of absolute rule, and here I am, appearing to break that rule.”
Not at all. But what you are missing is that once a variable is part of the causal chain, it is very much of interest. If we are studying the use of aspirin in the prevention of heart attacks, it becomes silly to call “the inhibition of platelet aggregation” a confounder. Similarly for your fertility example. If the frequency of sex causes a change in fertility, it wouldn’t occur to us to dismiss it as a confounder. Instead, this would be a very interesting discovery.
It isn’t that you are breaking a rule, it is that you are failing to recognize what it is we are looking for.
HelenSan:
“What I hope you or anyone on this board can “get” is that I was taught differently in graduate school, and so was my husband and other scientists we know. What we have are different “scientific” cultures, each with its own “rules” of what the terms mean, how to use statistics, and what constitutes scientific rigor. Who’s right? Well, I think my way is right and medicine is wrong to use so few controls, and you think I’m ignorant. So there we are.”
That has been my point all along. Whatever you were taught in graduate school left you unable to understand the scientific rigor of medicine. It makes no sense for someone to pretend that their inability to understand scientific rigor in another field means that that rigor isn’t there. For example, I don’t pretend that the field of chemistry isn’t rigorous because their experiments aren’t performed blinded.
HelenSan:
“You say it is not possible for variables such as recreational drug use or socioeconomic status (SES) to be confounders because the technical use of the term “confounder” requires that (1) they would have to be correlated with both the IV and DV, and (2) they cannot be part of the causal chain.”
No, I’m saying that if drug X causes a change in recreational drug use or in socioeconomic status, we would be disinclined to dismiss this effect as a mere ‘confounder’.
HelenSan:
“(1) Imagine an RCT with poor and rich subjects assigned in equal proportions at the outset of the study. A bunch of subjects drop out of the study in the middle, and it turns out most of them were rich and most of them were in the placebo group. When they did, an SES bias occurred in the study that wasn’t there at the beginning. The placebo group is now comprised a higher ratio of poor subjects, and it turns out they had poorer diets and less healthy lifestyles than the study group. High SES, in this situation, becomes positively correlated with both the study group and better outcome. The DV could be reflecting the SES bias rather than the true effects of the study drug. Therefore, SES is a possible confounder.”
Again, this isn’t classified as “confounding”, it is classed as a bias introduced by drop-outs. To pretend that it is a confounder means that you miss the opportunity to deal with the issue in a more appropriate manner, or even recognize the extent to which it invalidates conclusions that can be drawn from your results. This also DOES NOT represent an example of a confounder that is part of a causal chain. Even if the rich people dropped out because they realized they were probably not getting the active drug and were able to afford an alternative, then the ‘confounding’ factor would be “recognizing that one is assigned to the placebo group”, which is already a well-recognized issue – one whose measurement/detection is often included in the study design.
You haven’t raised any concerns that are relevant or aren’t already known and measured. It just seems that you are unable to recognize scientific rigor in medicine, because as you earlier said, you were not taught the skills to do so. But it seems a bit silly to chastise us for your unfamiliarity.