Articles

Evaluating Treatment Claims: A Primer

TrueFalse2

I recently wrote about the claim that acupuncture can improve vision in patients with macular degeneration. In response, I received this e-mail:

At Discovery Eye Foundation we have an education and outreach program for people with age-related macular degeneration, the Macular Degeneration Partnership. We are constantly getting calls from people who have heard of a new “cure” or a way to stop their vision loss. It is always hard to interject reality into the conversation and hear the hope leave their voice, replaced by frustration or despair.

The e-mail suggested I write an article providing guidelines for consumers to help them evaluate the validity of treatment claims for themselves. On SBM we are constantly stressing the need to apply critical thinking to what you read, and the many pitfalls to be avoided. I’ll try to synthesize some of the principles into a handy list of questions.

What kind of evidence is there?

  • If the claim is based on nothing but testimonials, STOP RIGHT THERE. You can forget about it, or at least file it away until there is better evidence. Anecdotes are not evidence; they only serve to suggest promising avenues of research. Science is the only reliable way to determine if a treatment is safe and effective.
  • Is the claim based on a gold standard randomized, placebo-controlled, peer-reviewed study or some lesser kind of evidence like case reports? Was it published in a reputable mainstream medical journal? (If you’re not sure how reputable the journal is, you can look it up on tables of “journal impact factors.”)
  • Keep in mind that half of all studies are wrong. There are many factors that can lead to error. Preliminary or pilot studies that are positive are frequently followed by better, larger studies that are negative. We can never rely on one study without confirmation.
  • Was it a meaningful clinical study in humans? In vitro lab studies and animal studies may not be applicable to humans; if promising, they must be confirmed in good human studies. If it was a human study, did it show meaningful outcomes that made a real difference, like a reduction in heart attacks, or did it just show an improvement in lab values or risk factors?
  • Were there 10 subjects or 300? Large studies are more trustworthy than small ones. The fewer dropouts, the better.
  • Did it use an appropriate placebo control that subjects really couldn’t distinguish from the active treatment?
  • Have other studies found similar results? Are there any studies that show the opposite? You can search PubMed and look for them.

Who is making the claim?

Is it someone who is likely to be biased? Is it someone who provides the treatment or sells the product? Is it someone with expert medical knowledge or someone like the schoolteacher who claimed she had invented a cure for the common cold? Does the person have a good reputation, or a track record of making questionable statements? Is it someone who quotes or associates with unreliable sources like Mercola.com, the Weston Price Foundation, or the Health Ranger? Quackwatch has a useful list of nonrecommended sources of health advice.

Where was the claim reported?

Real medical breakthroughs would be headline news. It’s not likely you would first hear about a cure for diabetes on an afternoon talk show or a Facebook page. Has your doctor heard of it? Are mainstream doctors recommending it? Is it covered on professional medical websites like the American Academy of Pediatrics or disease-focused websites like the American Diabetes Association? Is it sold only through multilevel marketing schemes?

What kind of language is being used?

Is it a sober factual report with caveats, or is it full of hype and buzzwords like “miracle,” “natural,” “known to the ancients,” “quantum,” “amazing,” “revolutionary”? If it really worked, advertising gimmicks wouldn’t be needed to sell it.

Does Gwyneth Paltrow swear it worked for her?

Testimonials can be very passionate but they are notoriously unreliable. Getting better when you use a treatment doesn’t necessarily mean you got better because of the treatment. Symptoms can fluctuate, diseases can resolve without treatment, and placebos can fool people. Perceptions can be wrong (think of optical illusions), the meaning of true perceptions can be misinterpreted, and memories can be inaccurate. Every snake oil salesman has reams of testimonials, and through the centuries there were testimonials galore for bloodletting to balance the humors. People frequently come to believe bogus remedies have worked for them. Barry Beyerstein wrote a classic article about that; it’s essential reading.

Does it make sense?

Is there a plausible mechanism of action? If someone claimed that standing on your head and whistling Dixie would cure diabetes, I think you would be skeptical. If it claims to work by a mechanism incompatible with known scientific principles, the level of evidence would have to be extraordinary for it to outweigh all the evidence those scientific principles are based on. Homeopathy’s claim that water can cure by remembering long-gone molecules, even after the water has been dripped onto a sugar pill and allowed to evaporate, would require extraordinary evidence indeed. If it’s a new antibiotic that is related to an old one, an ordinary level of evidence would suffice.

Is there a double standard?

Are they asking you to accept a “natural” or “alternative” treatment on the basis of the kind of evidence that you wouldn’t want the FDA to accept for allowing marketing of a prescription drug? There is only one science and only one standard of evidence.

Does it sound too good to be true?

Then it probably is too good to be true. Does it promise to cure a hitherto-incurable disease? Does it promise you can eat all you want and still lose weight? Does it promise there are no side effects of any kind? Does it remind you of the spiel of a used car salesman or a TV infomercial? Caveat emptor.

Who disagrees and why?

This is the most important question you can ask. It is rare for 100% of people to agree on anything. If you can find someone who disagrees, you can examine the reasons given for both opinions, and it will usually become obvious which side makes more sense. If you can’t find anyone who disagrees, it might be because it’s too new or because no scientist has taken it seriously enough to bother writing about it. In that case, withhold judgment and keep checking until someone does disagree.

Hope springs eternal, but true hope is better than false hope

If you are a desperate patient, it’s only natural to grasp at any straw of hope; but when the evidence is insufficient, the reasonable approach is to withhold judgment and wait for better evidence. You might think, “If it works, I don’t want to wait” but history teaches us that the great majority of these things don’t pan out. It might not do any harm, but then again it might; there might be adverse effects that haven’t been identified yet, it might raise false hopes only to dash them, and if nothing else it might waste time and money or interfere with getting more appropriate care. When you take an inadequately-tested medicine, you are essentially offering yourself as a guinea pig in a haphazard uncontrolled experiment that doesn’t even keep records. Of course, that’s your privilege; but I hope you would do it with your eyes open, with a realistic understanding of the state of the evidence.

Note: We sometimes take requests, as I did for today’s post. If you have done your best to apply these suggestions and are still uncertain whether the claims for a treatment are valid, drop us a line.

Posted in: Science and the Media

Leave a Comment (84) ↓

84 thoughts on “Evaluating Treatment Claims: A Primer

  1. Ruth says:

    I teach a summer camp on the chemistry of consumer products (like food, toiletries, and toys) for high school students. This nearly perfectly summarizes what I talk to them about on the very first day about finding reliable resources to trust when making decisions about products or ingredients!

    1. Aaron says:

      Ruth – I am an atheist, but: BLESS YOU. :-) I hope when my girls are older there will be summer camps like this in my area (near Chapel Hill, NC).

  2. DavidCT says:

    What is truly discouraging about the spread of quackademic medicine is the way it is treated by otherwise reliable journals. Dr. Crislip has pointed this out in the case Annals of Internal Medicine. It is peer reviewed but the peers for the altmed section are obviously quite different than those for the rest of the journal. As a result your suggestions for evaluating sources is more difficult than it should be.

  3. Dan Clarke says:

    “Did it use an appropriate placebo control that subjects really couldn’t distinguish from the active treatment?”

    That alone would undermine most of the studies used to hype interventions like CBT.

    1. KayMarie says:

      Should I leave? I was in a pretty large CBT trial as a subject. Happened to do me a world of good, and over the long term. I do wonder how good any of the sham treatments are, but at least some of the features I’ve seen of clinical trials (from both sides of it) the shams do control for most of the increased attention and being listened to you get in a trial that most of us typically don’t get in the MD’s office. I sometimes think a lot of the CAM effect is that you get an hour’s worth of attention rather than 10 minutes that most of my docs are allowed to spend with me.

      1. Dan Clarke says:

        There is still often a problem with the model being provided to patients. eg: If patients in the CBT arm are being encouraged to believe that they have greater control over their symptoms, then there are more likely to be problems with response bias. Also, a lot of trials for psychological and behavioral interventions still just compare with TAU.

        1. mouse says:

          @Dan Clarke -and all – Not sure of a good place to post this response. but-

          Folks might be interested in a couple of articles written by James Coyne for SBM on claims for psychotherapy and cancer. I think he is stating some similar concerns.

          http://www.sciencebasedmedicine.org/the-mind-in-cancer/

          Click his name in that article and you will get his archive. He also has a blog, but I haven’t read it yet.

          James Coyne is a clinical psychologist/professor/skeptic who focuses on criticizing psychotherapy research with an eye toward improvement.

          I see you (Dan Clarke) are receiving is a lot responses in this thread.

          I hope commentors can keep in mind that criticism does not nessacarily imply that one is against a particular intervention all together. Often, improvement of a system is only effected through criticism.

          I hope that you keep in mind that an intervention being hyped in some papers or analysis does not mean it is ineffective or unproven in all applications. The example of thyroid hormone replacement comes to mind. But there are probably better ones.

          1. Dan Clarke says:

            Thanks.

            “I hope commentors can keep in mind that criticism does not nessacarily imply that one is against a particular intervention all together. Often, improvement of a system is only effected through criticism.”

            I hadn’t really realised that this would need saying.

            Of course, examples of hype around CBT does not mean that CBT is never effective, but that CBT can be effective also does not mean that we should be less concerned over the hype that does occur.

            1. mouse says:

              Dan Clarke “I hadn’t really realised that this would need saying.”

              Perhaps, but this is a blog where a disproportionate number of new folks initially post talking about problems in pharmaceutical company research (or the like) who eventually revel themselves to be very extreme alternative medicine proponents who are entirely against anything pharmaceutical but recommend things like flower remedies for cancer or organic food for rheumatoid arthritis.

              So in an altered reality, sometimes people have to be reminded that normal people exist.

              Thanks for the softening of the response.

          2. Dan Clarke says:

            PS: My first reply now looks a bit blunt, but there’s no edit – sorry if it seemed rude.

    2. Cervantes says:

      Well, truly undetectable placebo control is impossible for some kinds of interventions. But it isn’t necessary if there are objectively measurable outcomes. If people drink less alcohol or stop smoking or whatever, then that happened. That they weren’t blinded to the intervention is beside the point. And if subjective relief is what you’re after, who even cares if it’s a placebo effect — in the case of CBT, what would that even mean? I want to feel less social anxiety, and now I do. So I’m happy.

      1. Dan Clarke says:

        Yes – or body weight for some eating disorders. When trial’s don’t need to rely on self-report measures there’s much less reason for concern. It’s important not to assume that, for example, improvements in questionnaires assessing social anxiety effect real improvements in patients social anxiety though – as in non-blinded trials things like response bias can serve to seriously distort results.

    3. Jesse says:

      Actually you could compare CBT to a placebo control that meets Dr. Hall’s standard. The control group could be patients who only receive unstructured “talk therapy” sessions versus structured CBT sessions.
      Also to my knowledge CBT is a well respected and researched intervention.

      1. KayMarie says:

        In the study I was in the sham was educational type sessions. They reviewed symptom diaries and got information and attention but none of the intervention.

        CBT generally seems to, I thought, have a pretty good track record research wise. You just really have a hard time blinding the patient and the person interacting with them can’t be blind as they have to know which intervention they are doing.

        1. e-rook says:

          For some indications of CBT, there may be objective measures that are more reliable than self-report response to questionnaires. An activity monitor can measure sleep; waking cortisol (elevated cortisol is associated with depression / anxiety) can be measured, dexamethasone suppression test can be administered (glucocorticoid insensitive people don’t respond, an indication of several disorders). A way to introduce some aspect of blinding is to have an evaluator who is not the intervener to pre/post assessments for symptoms using a standardized interview like the CIDI (composite international diagnostics interview), which may be computer-guided for the evaluator.

    4. WilliamLawrenceUtridge says:

      Oh good, someone else that claims, without evidence or citations, that CBT is useless. Well thank Dog there was a stranger on the internet to enlighten me, now my questions about therapy are resolved.

      1. Dan Clarke says:

        You’re replying to me, and yet I never said that CBT is useless.

        1. WilliamLawrenceUtridge says:

          You also didn’t provide any citations.

          1. Dan Clarke says:

            I’m not too sure what you want a citation for, as I didn’t say that CBT was useless. I detailed some problems with results for a CBT trial (without a control able to account for problems with bias ) being hyped here: http://www.bmj.com/content/347/bmj.f5963/rr/674255

            1. KayMarie says:

              Your initial comment was vague enough that it sounded like you could have though all studies on CBT are hyped, meaning none of the results indicate CBT works (aka it is useless).

              If you wanted to discuss a specific paper (or get clicks to something you wrote) why not lead with that rather than make us drag it out of you. It seems more honest if you are upfront about it. :-)

              1. Dan Clarke says:

                I hadn’t planned to discuss a specific paper, or get clicks. I just read the article and wanted to quickly raise the point that seemingly mainstream interventions like CBT can have similar problems with poor research and hyped results to CAM. We should be equally rigorous in the critical examination of all medical claims.

                It really seems quite a stretch to see my initial comment as claiming that all CBT is useless.

              2. KayMarie says:

                Dan, just in your haste you made a statement so vague it could mean anything, and seemed to imply much more than you intended.

                I’m glad you eventually clarified what you meant.

          2. WilliamLawrenceUtridge says:

            1) That’s not a citation, that’s a rapid response to a journal article you presumably didn’t like (I didn’t read it). It’s about as credible as a blog comment.

            2) I wasn’t aware that you were a CFS patient objecting to a study of CBT for CFS. That makes me even less interested in continuing this conversation.

            1. Dan Clarke says:

              1) You’ve still not said what you want a citation for. The comment I linked to included 18 different citations, which you’d have realised if you’d read it.

              2) What interesting prejudices and assumptions.

              1. WilliamLawrenceUtridge says:

                Having had really, really energetic discussions with CFS patients, extending for months, every single day, for hours each day, with claims of absolute certainty about the etiology and treatment of CFS, I’m quite comfortable with my prejudices and assumptions, thanks.

              2. Dan Clarke says:

                When someone’s happy with their prejudices, not much can be done about it. When dealing with your patients though, you should at least acknowledge that this is the case, rather than pretend that you are working to ensure that your beliefs are supported by the evidence.

              3. DW says:

                The problem here is the word “prejudice”; it refers to “pre” judging, or making judgments about something before you get all the facts.

                It sounds like William is not guilty of “pre” judging, rather he is “post” judging, also known as drawing conclusions from evidence.

              4. Dan Clarke says:

                Why do you think it sounds like that?

                He said “Having had really, really energetic discussions with CFS patients, extending for months, every single day, for hours each day, with claims of absolute certainty about the etiology and treatment of CFS, I’m quite comfortable with my prejudices and assumptions, thanks”, “(I didn’t read it)”, “I wasn’t aware that you were a CFS patient objecting to a study of CBT for CFS. That makes me even less interested in continuing this conversation”.

              5. WilliamLawrenceUtridge says:

                Oh, I’m not a doctor, I don’t see patients.

                Also, one should recognize that even though one might find the idea offensive and somehow insulting, the reality is that many people with CFS have found CBT helpful – and to discount this because of personal prejudices about one’s preferred beliefs regarding etiology is pure motivated reasoning.

              6. Dan Clarke says:

                I wrongly assumed that if you were having discussions with CFS patients every day, you would be a doctor. I apologise – your prejudices are of rather less concern if you do not have authority over others, even if it is still an unreasonable way to approach people.

                Your second paragraph seems to have nothing to do with anything discussed here, and I’m not sure if it reflects some unfounded assumptions of your own about me.

              7. WilliamLawrenceUtridge says:

                1) A massive number of CFS patients have strenuously objected to even the idea that CBT can help with CFS, presumably because that would make their suffering “only in their head” and thus less worthy of respect.

                2) You are a self-identified CFS patient who objects to CBT.

                3) I put 1 and 2 together, along with 3.5

                3.5) Many of the CFS patients I have argued with have been strongly argumentative and disagreeable, so I enjoy baiting them. It’s cruel, but everyone needs a hobby.

              8. Dan Clarke says:

                That’s a poor attempt at an argument.

                Even if 2 were correct it really wouldn’t add up to anything. 1 seems to largely be your own presumptions about the motivations of others. As for 3.5: you enjoy cruelly baiting people with health problems, yet find that it is they who are disagreeable?

                Maybe CFS patients often disagree with you because what you say about CFS is not terribly well thought through?

              9. MadisonMD says:

                I thought Dan’s original point is that CBT does not have adequate controls due to difficulty of blinding. Although it is mainstream it would seem to fail at least one criterion set forth in the post by Harriet.

                One other point of contrast however could be that at least CBT has plausibility given what is known about human psychology. So it may be acceptable to adopt it with less stringent blinding of RCT. I would have to accept Dan’s point that at least some of the effect could be placebo.

                To me it looks like internet communication blew this exchange of ideas among reasonable people out of proportion.

              10. mouse says:

                WLU “3.5) Many of the CFS patients I have argued with have been strongly argumentative and disagreeable, so I enjoy baiting them. It’s cruel, but everyone needs a hobby.”

                William – You realize that’s profiling, just replaced the words CFS patients with the word blacks, muslims, women, men or bipolar patients.

                So you’re like the SBM version of stop and frisk?

              11. KayMarie says:

                It isn’t just CFS patients. Seems like there is a group people who are miserable for any condition who get in that mode where anything that sniffs like “it is in my head” just sets off a whole chain reaction of anger and arguments in a reliable enough way.

                Even, I’ve found, when you aren’t even trying to set them off they seem primed to go into the same set of rants. At least if you are trying to set it off you know what is coming. But I’ve found all too often you get the same response to trying to be supportive, generous, kind and helpful.

                I’m glad when offered the CBT trial I decided to trust the research up until then and the peer-reviewed process that led up to them getting the grant for the study. Even though it can’t be done in a double-blind way. FWIW they did some physical tests and pre-CBT there were parts of the test we did not complete because I could not tolerate the pain for the required length of time. After CBT that part of the test was really no big deal so it wasn’t all just I was tricked into saying I felt better.

              12. mouse says:

                KayMarie”It isn’t just CFS patients. Seems like there is a group people who are miserable for any condition who get in that mode where anything that sniffs like “it is in my head” just sets off a whole chain reaction of anger and arguments in a reliable enough way.”

                For example me. When my husband and I were getting treatment for unexplained infertility, it wasn’t unusual for people or magazine, news, self help articles on overcoming infertility to suggest that my state of mind was at least partly causing the infertility. “If you would just relax – have you tried relaxation techniques? Depression can cause infertility, consider CBT for depression.” Yada, Yada, and damn I tried, I guess that’s the “Hope springs eternal that HH was talking about” but nothing doing. As a consequence I do have a knee jerk strong skepticism toward many mind/body approaches. Precisely because looking into it, I found many of the claims to be inflated and I found those inflated claims, to be pretty emotionally devastating.

                Luckily*, I did see a therapist, who was helpful, in the end. The irony here is that one of the ways she was helpful was in her honest, nuanced assessment of the limitations of therapy (cutting through the “hyped” mind/body approaches which a lot of journalism focused on) which helped me reach the conclusion that adoption was a better approach for me.

                To me, considering that CBT has limitations and is not adequate treatment for many conditions, including both organic and moderate to severe psychiatric one, along side the fact that journalism today often lacks the nuance and overstates the results of their study de jour, I am not surprised that some patients feel that CBT is hyped.

                *I would add that there are therapists who practice CBT and are very aware of the limitations. They can be a good resource in helping a patient to find the additional help they need. But I am also acquainted with people who have seen therapists who actively discouraged medical treatment (such as medication for bipolar) in favor of therapy – which I don’t think was appropriate or beneficial for the patient. So when I say “luckily” I really do mean that.

              13. KayMarie says:

                I agree that especially in “journalism” reports of a lot of things are more about the hype so they can sell ads rather than good reporting. Which I don’t consider actual journalism.

                The one I run into is the idea that if the [fill in physical complaint] went away the [fill in the mind symptom or stress] would magically go away all by itself. Along with the only thing that can possibly be treated with CBT is the mind symptom.

                Often the secondary issues don’t magically go away after whatever triggered them goes away. Once triggered they last on. What I found interesting in the paper about the study I was in was that often the CBT worked better for symptoms when there wasn’t overt depression or anxiety going on. So it isn’t just treating depression, but seems to work on symptom reduction.

                I’m pretty cautious about hype, but when you go from needing a lot of medication (once I was off the extended release I was taking the short acting version 7-8 times a day and that was with a controller medication on top of the symptom reliever) to one or less of the short acting reliever and 1/2 the dose of the controller in a few months. Particularly because the vast majority of the pain relief happened the same week a s a therapy breakthrough.

                Could be coincidence, but that it worked for over 2/3 of the people in my treatment arm tends to make me think my dramatic improvement may not have just been spontaneous remission or placebo effect.

                At least on the scientific/researcher end of it I have never heard anyone claiming it is the end all be all cure of all things mental and physical. Just for some conditions and symptoms it happens to be quite effective. Often more effective in combination with medications than when done by itself. What the “journalists” do with the information is whole ‘nother story.

                That being said, I usually try to not tell anyone, even in the support group, how well CBT worked for me. Even just saying it worked for me is sometimes more than enough to get way too much anger and crap thrown at me than I usually want to deal with. I try to ignore it as it isn’t really personal, but some people won’t stop until they get a reaction. Thank goodness that online community has an ignore feature so I can at least make it so I don’t have to see it once I get tired of it. In the end I think I still help more people than I bother, so I stay and try to do the best I can to help others.

              14. Dan Clarke says:

                Pardon the long post replying to a few things at once.

                1) Judging from some of what we’ve seen here, it seems quite possible that at least some examples of what people saw as unreasonable CFS patients seeing CBT as ‘all-in-the-head-ism’ could just have been misunderstandings.

                I think that people deserve to be treated as individuals and for the points and arguments they put forth to be responded to on their merits, rather than with a misguided ‘I know your sort’ mentality.

                2) e-rook and others mentioned objective vs subjective outcomes for non-blinded trials: In the comment I linked to I had cited some research on this relevant to CFS that could be relevant to this discussion.

                The first trial for CFS that found CBT did not lead to improvements in activity levels led to the developers of CBT for CFS claiming that this showed the treatment had not been carried out properly, as “the primary aim of treatment is to restore activity and function” and “If a patient completes the program, he or she must have increased their activity”. Now that a number of non-blinded trials have shown that CBT leads to improvements in self-rated disability, but not objectively measured levels of activity, there has been a shift to arguing that it is the self-rated measures which should be seen as most important, and it is these which are used to inform patients about CBT’s effectiveness. I think this is a pretty questionable approach.

                For conditions that do allow more objective outcome measures it is important that they are made use of, but also, I think that the disparities between objective measures of disability and subjective self-report measures in non-blinded trials indicate that we should be cautious with the claims made about treatment efficacy when blinded trials or objective measures of outcome are not available. Patients should be informed about the danger of biased results when the costs and benefits of an intervention are being discussed with them so that they can make their own informed choices. The problems which lead to exaggerated claims being made about the value of CAM will also affect similarly designed trials for more ‘mainstream’ interventions.

                3) MadisonMD mentioned plausibility (& thank you for recognising my initial point): I just quickly wanted to add a concern that, in areas where we don’t have much understanding or solid evidence, almost anything can be seen as plausible. That implausible CAM interventions are able to generate results which can seem to indicate efficacy has helped highlight how trials can be biased, and the need to apply rigorous critical thinking. I think it’s important to apply this equally to all proposed interventions.

              15. KayMarie says:

                While I often try to treat people as individuals, there are some things that tend to be very common human failings and strengths and it really isn’t surprising when people exhibit either of them. Recognizing some of the patterns can be helpful in deciding whether you want to go down that rabbit hole again and get attacked the same exact way again or not.

                I don’t know the CBT for CFS lit that well but I do know the study I was in for another conditions there were measurements that were objective as well as subjective and the objective ones got better as well as things like how much pain you are in which is always subjective.

                FWIW I don’t think the informed consent needs to be different level for CBT than for any other treatment. When I have entered CBT or biofeedback or other mainstream treatments that can be tarred with the alternative brush fairly easily I didn’t get unreasonable expectations of benefits or risks. Pretty much got the same kind of we have some data that indicates it might work and if you are willing we will enroll you in the clinical trial/give you the referral. Maybe it is because I’m a scientist they feel more able to describe the risks, but usually they seem to be the standard protocol, not something special I and only I get.

                Now there are bad doctors out there, but I don’t see them (at least the ones I’ve seen) being anything than honest about any of the treatments I got. Even if I could have interpreted it as “all in my head”….well I guess the migraines were all in my head, but not in the I think my doctor is telling me he/she thinks I’m imagining my symptoms kind of way.

                Quality of life vs more objectively measurable symptoms is always going to be one of the questions. Does it matter if it makes the target symptoms objectively better if it decreases your quality of life too much, or is a treatment worth doing if it only improves the quality of life. Now with the second a lot depends on risk. If there isn’t a lot of risk from trying it and it makes your quality measures much better that doesn’t mean it is automatically bad medicine that should never be recommended. Now it does need to be honestly represented to the patient. Sometimes illnesses that only make you wish you were dead are a lot worse to deal with than the ones that actively try to kill you.

                For me quality of life also matters, but your mileage may vary and you may be unwilling to do anything that might only improve quality of life and not change some number on some test result. You should be informed of what kinds of effects are likely, and the stats for them. As well as a reminder that population statistics mean nothing at the level of the individual. It will either work or it wont.

              16. Dan Clarke says:

                If you’re using that sort of profiling out of a concern over being attacked (criticised?), surely it would be worth just avoiding whatever topic you feel uncomfortable with.

                I don’t think informed consent should be different for CBT than anything else either. I think that patients should be informed of potential problems with response bias for all medical interventions that’s efficacy is measured using self-report measures in non-blinded trials.

                The comment I linked to cited some specific examples of the results from a particularly large and important trial being misrepresented. It also seems that there is a greater willingness to put a ‘positive’ spin on claims of efficacy for patients suffering from certain conditions, and that again, this sort of profiling is unreasonable.

                Quality of life is certainly important, but I’m not sure that it’s safe to assume the questionnaires we use are reliable measures of quality of life, particularly not when it comes to measuring the impact of a certain interventions on a group of patients. It seems like this is becoming an increasingly important political point, and I think that there’s a danger that this gives too much power to those who claim to be able to measure Quality of Life, or even just decide what ‘Quality of Life’ should be. It’s quite possible that we both have quite different desires for our own lives, and views about how the quality of our lives should be measured and assessed.

              17. KayMarie says:

                You sound like you assume no doctor EVER would give appropriate informed consent for something like CBT. I mean they gave adequate consent for the study I was in which passed the IRB so it can be done.

                Usually MDs, in my experience are more likely to underplay how well a treatment may work because they are supposed to tell people the uncertainties and avoid the hype (I’m looking at you Dr. Oz). One of the reason sCAM sometimes works well is the practitioners is 110% certain beyond all reason that his treatment will work for you totally and completely. You give a person certainty and it tends to increase their belief they got better.

                Generally most surveys and measures of quality of life have to be vetted and well studied before you even suggest using them in studies and usually more so for clinic use. If you tried to put a measure that had little or no evidence for it in a grant you would not get the grant. There are some pretty standard protocols for developing these kinds of metrics and a fair amount of understanding on how they work. Now what journalists or reporters do with the data (which is almost always an exaggeration what the researchers said because they are trying to get eyeballs on their spewing) can generate some hype that most people who read the papers, not the headlines can correct for.

                Now I know QOL is squisher than some measures, but probably not that much squisher than asking people to rate their pain on a scale from 1 to 10. My 7 may be someone else’s 10. Heck, when I had to rate my pain every week for the CBT study I was in I found my 7 at the end of the study when my overall pain was a lot less was very different from my 7 when the study started. I did mention that and there apparently are some ways to deal with that sort of thing as they expect it.

                I guess the question on attacks is how to help those who actually want help while avoiding those who are so miserable they will see any suggestion or information as an opportunity to take out their anger on someone. Sometimes you profile people and avoid them when they click the safety off rather than wait to see if they start shooting at anything that moves.

              18. Dan Clarke says:

                “You sound like you assume no doctor EVER would give appropriate informed consent for something like CBT.”

                Would you be able to quote the bit that led you to believe that. I don’t see how anything I wrote could be interpreted in that way.

                “Usually MDs, in my experience are more likely to underplay how well a treatment may work because they are supposed to tell people the uncertainties and avoid the hype”

                I cannot comment on your experiences, and know that there are doctors who take their responsibilities around informed consent seriously, but I did provide some clear examples from published papers of senior clinicians making exaggerated claims for the benefits of the interventions they are paid to provide, and that concerns me.

                I think that the discussion around our ability to reliably measure individual’s quality of life might be a complicated side issue that would be best left for another time.

                I don’t think I fully appreciate your concerns about being ‘attacked’ in an on-line discussion. You use the imagery of gun violence, but I assume you’re just referring to people typing things. If one is cautious, rigorous, moral and reasonable in the claims one makes, people’s responses don’t really matter. I would have thought that any sort of prejudicial profiling of others would be likely to make things worse rather than better.

                To me, it seems unsafe to assume that those who ‘attack’ you do so because ‘they are so miserable they see any suggestion or information as an opportunity to take out their anger on someone’. Maybe it’s the result of misunderstanding? Maybe some of the claims you make are ill-informed? Maybe they feel that you are profiling them and treating them unreasonably? Who knows? If they are ill and a bit cranky, I think it’s best to be cautious in making assumptions about them which could be seen as disparaging.

            2. KayMarie says:

              ” think that patients should be informed of potential problems with response bias for all medical interventions that’s efficacy is measured using self-report measures in non-blinded trials.”

              Seems to imply that you think this may not be done.

              I was in a study where the results were not over-hyped or over-represented, but I wonder if any positive result is too much hype for you. Generally researchers tend to be fairly circumspect about the limitations of what they publish. There are exceptions, but that you got a grant to support your research is not the same as being a paid shill for Big Psych.

              For some reason to me it seem that to you my experiences and opinions do not matter and are inconsequential, or at least massively wrong, so I’ll just stop now. If you feel profiled as one of those individuals who cannot tolerate that other people may have a different viewpoint as upsetting and I apologize in advance, but I really don’t need to be dissected by you any further given I am not allowed to have interpretations of your typing. Take care.

              1. KayMarie says:

                I realized I did have a couple of points to make before I feel complete and ready to move on.

                Sure, I’ve volunteered for two decades to help patients understand the science behind their disease, figure out how to work with the doctors, develop self management plans, and explore the treatment options just to be a jerk or whatever interpretation of me is behind your typing.

                (and yes I’ve gotten faster at picking up on the clues in the typing that someone would not appreciate certain types of support, call that profiling if you will, but if I clue in that I’m going to upset them more than they already are I just leave them alone, if that makes me a jerk and a terrible person, so be it)

                FWIW there are bad studies, but that doesn’t mean all studies are bad. At least for my disorder most doctors won’t even recommend CBT even though it has better data than a fair number of the other treatments they prescribe. Far from pushing it on people and hyping it up they seem as leery of bringing it up as I am most of the time. I’ll occasionally mention it as an option but really it isn’t worth the headache I get it I dare to lead with it or be too vocal about how it helped me.

              2. Dan Clarke says:

                “For some reason to me it seem that to you my experiences and opinions do not matter and are inconsequential, or at least massively wrong,”

                There’s nothing about the experiences you’ve described which in any way challenges anything I’ve said or believe, and also, there are a lot of anecdotes on the internet. I don’t really know what sort of response you wanted, or how consequential you think your experiences should be to me. It does seem that you keep assuming the things I write mean quite different things to that which they say.

                eg:

                I don’t see how:

                ”I think that patients should be informed of potential problems with response bias for all medical interventions that’s efficacy is measured using self-report measures in non-blinded trials.”

                could be taken to show:

                “You sound like you assume no doctor EVER would give appropriate informed consent for something like CBT.”

                “just to be a jerk or whatever interpretation of me is behind your typing. ”

                I don’t know if you’re a jerk or not, but I do think that you misinterpret what others write.

                “FWIW there are bad studies, but that doesn’t mean all studies are bad”

                Of course.

              3. WilliamLawrenceUtridge says:

                I was in a study where the results were not over-hyped or over-represented, but I wonder if any positive result is too much hype for you.

                Heh, I see what you did there, and I approve.

              4. Dan Clarke says:

                “” I was in a study where the results were not over-hyped or over-represented, but I wonder if any positive result is too much hype for you.”

                “Heh, I see what you did there, and I approve.”

                It did seem like the sort of thing you’d approve of.

              5. WilliamLawrenceUtridge says:

                Well when someone denies another’s personal experience because it doesn’t suit their preconceptions, I enjoy when this is pointed out in an entertaining way.

              6. Dan Clarke says:

                “Well when someone denies another’s personal experience because it doesn’t suit their preconceptions, I enjoy when this is pointed out in an entertaining way.”

                Could you try to quote a section which shows me denying another’s personal experience because it does not suit my preconceptions? I am quite confident that you will not be able to do so.

              7. WilliamLawrenceUtridge says:

                I’m interpreting.

              8. mouse says:

                WLU “I’m interpreting.”

                Interpretive strawman dance.

                Come on WLU, If you have a substantive disagreement based on real information and data relevant to the quality of CBT research, rather than vague interpretation and personal experience, maybe you should just go ahead and state it.

                At this point it looks like you got nothing.

              9. WilliamLawrenceUtridge says:

                I have very little but pre-existing prejudices, some reading into the comments based on them, and a total unwillingness to undertake the research. I’ll cop to that.

  4. Nancy says:

    I would also suggest that people use these rules for not only questionable practices and medicines but also FDA approved medicines used in off label situations. There seems to be an epidemic of doctors using any drug to throw at a difficult disease just because there is anecdotal evidence that it works. As a migraine sufferer I’ve had several doctors throw a myriad of drugs at my disease because they’ve read an article that it might work, with little discussion of side effects or drug interactions. Now I put real doctors through the same rigorous questions that I do other so called experts.

  5. EvilTwinSelf says:

    How about:

    “Is the claim about one specific disease/problem, or many different ones?”

    Perhaps this is partly covered under the other categories, but some of the most questionable products and treatments are claimed (by their proponents) to treat or cure a wide range of different and unrelated problems. I suspect few genuine medicines or treatments can make this claim.

    At the least, broad claims like this should set the alarm bells ringing that the person making the claim could be exaggerating (or completely making up) the benefits of the treatment, unless there is strong evidence for each part of the claim individually.

  6. Kelley says:

    I agree this is true for the vast majority of “miracle” cures. And yet, self-experimentation has an honorable history of advancing legitimate medicine. That’s why I don’t reject anecdotal (n=1) experience out of hand. In the field of diabetes and nutritional research, Dr. Richard Bernstein is one of my self-experimenting heroes (http://www.diabetes-book.com/book/mylife.shtml). An engineer before he became a doctor, he brings a very clear methodology to his experiments. Yes, there’s a book he’s selling, BUT his tone is no-nonsense. It’s not “I’ve discovered the truth and now I need to convince you.” It’s “here’s my story, the science behind my research, and here are the stories of the many patients I’ve helped; maybe this can work for you too.” And still his ideas haven’t been fully embraced by the mainstream (the American Diabetes Association, for example). So his results don’t pass at least two of the tests described here: selling something and not embraced by mainstream med. (I also realize that the article isn’t advocating for a 100% passing rate to the litmus tests it proposes.)

    Examples like Bernstein (and a few others I could name) are why I say critical thinking IS essential–even when you’re reading an article espousing the importance of critical thinking in evaluating a treatment. Don’t adopt an all-or-nothing attitude toward anything.

    Keep up the good work!

    1. KayMarie says:

      The question isn’t do you start with an n=1 experiment but is that where you stop and proclaim the revolution is at hand.

      1. Kelley says:

        Excellent point, KayMarie, and one I didn’t make clear in my post. N=1 is a starting point.

    2. WilliamLawrenceUtridge says:

      I’d respect Bernstein more if he took his “here is my idea” and did the actual hard work of running a clinical trial of some sort. Repeatedly testing anecdotes is far, far less useful to patients in the long term than actual clinical trials.

      1. MadisonMD says:

        Agreed. He was right about the benefit of tight glucose control but is was left to others to show the benefits. His low carb ideas could use some evidence but highly plausible for diabetes at least if not taken to an extreme.

      2. Andrés says:

        The low saturated/any fat advice was based on very shallow epidemiological evidence. No long term proof was asked for. You don’t need any direct evidence to use a glucometer and carbohydrate restriction* to try and keep your blood glucose levels under those that give rise to organ damage. In this case it is not prudent to wait for population level recommendations that are not going to take into account your personal reaction to meals.

        Just a couple excerpts from Dr. Bernstein’s biography:

        With Suther’s backing in the form of free supplies, by 1977 I was able to get the first of two university-sponsored studies started in the New York City area. These both succeeded in reversing early complications in diabetic patients. As a result of our successes, the two universities separately sponsored the world’s first two symposia on blood glucose self-monitoring. By this time I was being invited to speak at international diabetes conferences, but rarely at meetings in the United States. Curiously, more physicians outside the United States seemed interested in controlling blood sugar than did their American colleagues. Some of the earliest converts to blood glucose self-monitoring
        were from Israel and England.

        This “progress” was entirely too slow for my liking. I knew that while the medical establishment was dallying there were diabetics dying whose lives could have been saved. I knew also that there were millions of diabetics whose quality of life could be vastly improved. So in 1977 I decided to give up my job and become a physician—I couldn’t beat ’em, so I had to join ’em. This way, with an M.D. after my name, my writings might be published, and I could pass on what I had learned about controlling blood sugar.

        I don’t think it is fare to implicitly blame Dr. Bernstein of lack of commitment. In Science the burden of proof is on those making the claim. In Medicine the burden of checking some promising treatment should be on those entitled with the responsibility to care for the population health: professional societies?, government?

        *There are those that are not going to need to restrict them at all. There are those that are going to need to restrict high glycemic index ones. There are those that are going to need to restrict all of them. If you don’t measure it you don’t know in which group you are.

        Of course the effect of usual meals on blood glucose levels is not going to be static and should be checked sparingly but periodically for those not diabetic type I. WARNING: ANECDOTE ahead My postprandial at 1 and 2 hours after some meals has improved over a two years span. For example, octopus with boiled potatoes (3-4 medium ones) swimming in olive oil, two years ago 120mg/dl at 1 hour and 108mg/dl at 2 hours, one month ago 92mg/dl at 1 hour and 78mg/dl at 2 hours. As always I don’t know which of those things I am doing is responsible. I have excluded physical exercise on purpose. I will include high intensity one in the near future.

        1. Andrey Pavlov says:

          <blockquote.You don’t need any direct evidence to use a glucometer and carbohydrate restriction* to try and keep your blood glucose levels under those that give rise to organ damage.

          Andres, there is no theoretical lower limit to a glucose level that won’t cause end organ damage. The damage is caused by non-enzymatic glycosylation. There is no magic cutoff at which this simply ceases to occur. It is a smooth continuum down to zero. The damage is also always there, it is just a question of our ability to detect it. So it isn’t a question of a cutoff but where there is diminishing returns on keeping a lower blood glucose level and whether there are other harms from doing it and whether the difference is clinically significant or not.

          It is also not any single measure of blood glucose (regardless of which one it is) but the total AUC of blood glucose over the course of your day, week, and ultimately life that is important in this regard. Which is why we use HbA1c as a better marker for control than any blood glucose measurements.

          And all of this is why we have recently changed the guidelines for glucose control and use target HbA1c levels rather than target post-prandial or fasting glucose levels to measure control of diabetes.

          Ultimately, keeping yourself as an otherwise non-diabetic person below those levels of blood sugar are likely just way more effort than they are worth for all the above reasons.

          In Science the burden of proof is on those making the claim.

          Ah yes. Now if only you would actually apply that principle consistently and with intellectual honesty instead of applying it whenever it suits your argument you’d make some real headway.

          1. n brownlee says:

            Andrey, surely patient managed blood glucose levels, as measured by the diabetic with a glucose meter, are a necessary part of blood glucose control? I mean, that quarterly HbA1c is great- but my 3X daily self-check is the way I keep a handle on it.

            1. Andrey Pavlov says:

              @N brownlee:

              Yes, of course they are. That is how to best manage things on a daily basis, but primarily for knowing how well your medications are working (or whether they are working too much). It is first and foremost a means to titrate medications. But the real marker of how well you are managing the damage from hyperglycemia is the A1c. It is terrible for titrating medications though.

              I’ve had patients who have great fingerstick glucose readings and less than great A1c’s. And vice versa.

              1. n brownlee says:

                Type II diabetics can control the disease and tremendously slow the progress with tight blood glucose control; I did it for more than ten years. It does take fairly frequent self monitoring- 3X daily, usually- and it’s a restrictive diet- quite low carb – plus daily exercise. A quarterly average, while useful to the doctor, doesn’t much help the patient in learning self management.

                When I was learning how to manage the disease just with these measures, comparing notes with other diabetics, I was dumbfounded by how many people said, “No bread? No spaghetti? No way!” Many of them had never been told about even minimal carbohydrate restriction. And, required by my PCP to be instructed in diet management by a “certified diabetic counselor”, I was dumbfounded again by fifteen minutes of playing with a melmac dinner plate and some grubby rubber food. No weights, no measures, no explanation of the effects of food on blood glucose levels- just a dollhouse tea party.

                I do think it’s my responsibility to manage those aspects of the disease which I CAN manage. In diabetes, that’s quite a lot.

          2. weing says:

            There may be a role for continuous blood glucose monitoring for those on insulin to avoid hypoglycemia.

            1. Andrey Pavlov says:

              There may be a role for continuous blood glucose monitoring for those on insulin to avoid hypoglycemia.

              And yes, continuous monitoring with continuous infusion pumps (or that bionic pancreas from the NEJM article) are the holy grail of diabetes management.

              My real point to Andres was that the metrics he was looking at and referencing are not meaningfully translated into the outcomes he is hoping for.

              1. KayMarie says:

                For people using insulin there really is a need for routine self monitoring as you have to be careful with the dosing. The syringe isn’t hooked into the feedback mechanisms a functioning pancreas would have to deliver the appropriate dose.

                There are studies going on to see how much daily monitoring is needed for those who aren’t on insulin. Sometimes more testing is just more, not better. Some may depend on which patients can understand the data and use it to drive behavioral change vs how many it is just a bunch of sore fingers.

                I agree with the issue with diminishing returns. Often that is the problem in medicine. We know that if something is way out of whack there are more than enough benefits to justify the risks of the testing and treatment. But where is that point of diminishing returns where you end up doing more and more harm for each one person who may benefit.

      3. Andrés says:

        Dr. Pavlov said:

        Andres, there is no theoretical lower limit to a glucose level that won’t cause end organ damage.

        Perhaps. Since we all are going to die I don’t worry about damage not having any measurable effect on quality of life before it ends though.

        From the first paper referenced by Jenny Ruhl it seems that HbA1c is not a so clear measure of exposure to high levels of glucose:

        A total of 13 of the 107 patients had diabetes, whereas 36 (34%) had IGT, nearly three times the prevalence in age-matched control subjects (P < 0.01). OGTT was often elevated, whereas both fasting plasma glucose and HbA1c were normal.

        Of course it wasn’t a drug naive population so perhaps that’s the reason HbA1c levels as of Figure 1A was less predictive of neuropathy symptoms.

        Not that I dispute HbA1c usefulness (via Petro) —specially at the population level or to keep track through time of a patient— but I donate blood every three months so my HbA1c would give an optimistic AUC estimation. I don’t get it measured. When the intervention is diet not a drug postprandial measurement gives so much more actionable insight.

        Dr. Pavlov said:

        Ultimately, keeping yourself as an otherwise non-diabetic person below those levels of blood sugar are likely just way more effort than they are worth for all the above reasons.

        One of my two maternal uncles was a type II diabetic and died of a heart attack. The other one smoked a lot and also died of a heart attack. My maternal grandmother was hypertensive and she also died of a heart attack. Perhaps I will die of a heart attack too but it seems to me that I am going to dodge being diabetic, it’s been 8 years since I gave up smoking and my blood pressure keeps looking good. I have in check both my fasting and postprandial blood glucose in order to keep the AUC contained. I only measure them very sparingly though and only on my most common meals (I now skip green peas as a frequent almost-standalone meal because of results… I should measure again).

        Perhaps you have missed my second part to “In Science the burden of proof is on those making the claim”:

        In Medicine the burden of checking some promising treatment should be on those entitled with the responsibility to care for the population health: professional societies?, government?

        I don’t think it is Dr. Bernstein’s fault.

  7. Stella B. says:

    Another guideline for me is to avoid those claims that resort to the use of strawman arguments.

  8. Bruce A. Hamilton says:

    I guess I can safely assume that in the context of SBM, CBT refers to “cognitive behavioral therapy” and not “computer-based training” :-)

  9. Marcus says:

    “Did it use an appropriate placebo control?”
    Actually I think that always using placebo controls is a modern medical malpractice. For diseases that are already addressed by medication we should use the best known effective treatment on the control group. Why? Because we should be interested in knowing if we have found a better treatment, not only something that is better than nothing.

    1. Andrey Pavlov says:

      @Marcus. You are correct in your thoughts. The technical term is <a href="http://en.wikipedia.org/wiki/Clinical_equipoiseclinical equipoise

    2. KayMarie says:

      Depends on the study, but appropriate sham controls or appropriate standard treatment controls need to be in place if you can’t do a placebo control.

      A lot depends on the risk of doing nothing for that particular disease for the length of time of the study.

      There are comparative effectiveness trials but most typically they are between things we have shown are better than nothing in independent trials.

      I think a lot depends on how certain you are that the treatment you are comparing against has been adequately tested to where you can use it as a stand in.

      One issue is placebo rates vary from study to study and can even vary among the study staff (so which doctor in the study the patient saw) so there are downsides if you don’t know the placebo cure rate in this particular instance.

      As always it is a balance between risk and benefits and how you can minimize risks while getting the data that lets you know there actually is a benefit.

  10. WilliamLawrenceUtridge says:

    You realize that’s profiling, just replaced the words CFS patients with the word blacks, muslims, women, men or bipolar patients.

    So you’re like the SBM version of stop and frisk?

    I don’t see it that way, but the brain being a self-justifying machine, I of course wouldn’t see it that way. I see it more as stopping and frisking someone wearing a sandwichboard loudly and aggressively declaiming their sins and the need to adhere to the ways of the People’s Temple.

    Except with unnecessary confidence about equivocal science, rather than murder.

    1. mouse says:

      Really? Since when is stating an opinion on overreach in the interpretation of a study(ies) “aggressively declaiming their sins”? What sins you are imagining? Suggesting that some psychotherapy studies hype results? As I pointed out up thread, this is something that James Coyne has discussed on a few occasions and I don’t recall you pointing out his sins.

      In fact I urge you to check out the JC blog http://blogs.plos.org/mindthebrain/2014/06/10/salvaging-psychotherapy-research-manifesto/

      Where he says “Psychotherapy research is scandalously bad.
      Many RCTs are underpowered, yet consistently obtain positive results by redefining the primary outcomes after results are known. The typical RCT is a small, methodologically flawed study conducted by investigators with strong allegiances to one of the treatments being evaluated. Which treatment is preferred by investigators is a better predictor of the outcome of the trial than the specific treatment being evaluated.”

      If Dan Clarke is a sinner with a sandwich board. Then maybe James Coyne is the devil himself. :)

      1. KayMarie says:

        Well the complaint here was there wasn’t a placebo control in this one specific trial (and maybe CBT is the only one of the treatments tested that was hyped), but looking at the PACE study you could complain the previous studies didn’t do the placebo control properly, but this seems to be a comparative effectiveness or pragmatic research trial design.

        Which is asking a different question than is it better than placebo and one wouldn’t include treatment arms anything that had not already passed the is it plausible and is it better than doing nothing tests.

        Problem is if treatment A is better than placebo under one set of conditions (or usually a couple of studies which replicate the results but likely do not have completely identical measures or conditions) and treatment B is better than placebo under yet another set of conditions you can’t just say well A helped 53% and B helped 57% so B is always better than A.

        If the research question is which is better under this set of standardized conditions A or B you may not need to retest the or placebo. If A or B is not the current standard of care you probably will include a standard of care control using the same measures and conditions. So the is doing something new actually better than what we already do type of research question.

        So back to does there have to be a placebo control in every single study and if there isn’t….[fill in your interpretation or assumptions].

        Based on what grant announcements we’ve been seeing recently there are likely going to be more pragmatic and comparative effectiveness trials coming up and many of them probably won’t have a standard placebo. But for those studies to get funding you really need each of the potential arms to have the placebo controlled data in (and enough replication you can be reasonably certain the treatment has a plausible mechanism of action as well as evidence of efficacy). So they aren’t going to get funding to throw any random homeopathic remedy into the mix when comparing FDA approved medication A and FDA approved medication B.

        There is also a push to have more stakeholder engagement and patient oriented outcomes so I’m sure people will get even more certain that the wrong things were measured when you start including measures based on what actually matters to the people who suffer from the illness rather than just what you would pick based only on the medical lit.

        1. Dan Clarke says:

          I’m afraid that you did not really summarise what I had said accurately. Personally, I’d prefer to have you quote what I actually wrote.

          Also, minor points with regards to PACE: Adaptive Pacing Therapy was newly developed, and had never been tested in any form previously, while the Specialist Medical Care control was provided to patients in all arms, with additional treatments also provided in three of the four arms.

          I can only think of one trial which did use a placebo control for CBT for CFS, and that reported that CBT did not lead to additional improvements: http://www.amjmed.com/article/0002-9343%2893%2990183-P/abstract I can’t think of any which have tried to use a placebo control for GET.

          Also, for both these treatments and for other which are similar, it is very difficult to be confident that something like a placebo control intervention could fairly account for potential sources of bias. eg: Maybe the placebo in the study above did more to induce a positive bias, but the CBT intervention was genuinely helpful for some patients? There are certainly also ways in which one could also imagine CBT more effectively inducing positive bias than a placebo.

  11. Dan says:

    Hello Harriet,

    Just wondering if you have heard of this book: Vitamin K2 and the Calcium Paradox: How a Little-Known Vitamin Could Save Your Life. I’ve had several people ask about it and I am skeptical about its claims. http://www.amazon.com/Vitamin-K2-Calcium-Paradox-Little-Known/dp/0062320041 The book is written by a naturopath, and when I Google vitamin k Mercola’s site is one of the first, both of which seem like red flags to me. You had mentioned that you looked at comments when developing story ideas, so I thought I’d ask.

    Thanks for all you do,

    1. Harriet Hall says:

      I hadn’t heard of it. I am skeptical too. I’ll try to check it out.

    2. WilliamLawrenceUtridge says:

      If a naturopath recommends it, you can pretty much assume it’s nonsense.

Comments are closed.