Articles

Why Do They Do Studies Like This?

A recently published study claims to have shown that a proprietary mixture of velvet bean and Chlorophytum borivilianum improves sleep quality. The journal, Integrative Medicine Insights, is online, peer-reviewed, PubMed indexed, open-access, and it charges authors $1848.00 to publish their article. It advertises editorial decisions in 3 weeks and publication in 2 weeks after acceptance. I can see two reasons why authors might be willing to pay that much for publication: to speed the process of getting important research results out to the public, or because their research is poor quality and they know it would be rejected by other journals.

The quality of this study is unfortunately typical of much of the research on alternative medicine.

Description of Study

The full text is available for download here.  The title is “A Dietary Supplement Containing Chlorophytum Borivilianum and Velvet Bean Improves Sleep Quality in Men and Women.” They gave a proprietary supplement mixture to 18 young healthy subjects with self-reported impairment of sleep quality (defined as routine difficulty falling asleep, waking more than twice during the night, and awaking in the morning feeling tired) and had them fill out a questionnaire about sleep quality before and after the trial. They also measured heart rate, blood pressure, CBC, metabolic panel, and lipid panels.

Subjects were told there was a 50% chance that they would get a placebo, but no placebos were actually used; this was a clumsy, inadequate attempt to minimize any placebo effect, and they didn’t even think to ask subjects afterwards whether they thought they’d received a placebo.

They described the subjects’ characteristics: average age, height, weight, waist and hip measurements, % body fat, years anaerobic exercise training, hours per week aerobic exercise, etc. The relevance of some of these characteristics is far from clear, and the information is next to useless because it is presented in a table that lumps everyone together and only provides average values +/- standard error.

The study lasted 28 days. Subjects were told to take 3 capsules a day for 3 days and then take between 1 and 3 a day depending on their tolerance and preference, in an attempt to mimic real life use of dietary supplements. They asked subjects not to change their diet; they had them keep food diaries and did detailed calculations of everything from calories to selenium(!?) based only on self-reported intake. Subjects were told to continue their usual exercise but to abstain for 24 hours prior to the days when tests were administered.

Results

On the Pittsburgh Sleep Quality Index (PSQI), a self-reported questionnaire, they found statistically significant improvement in subjective sleep quality, sleep latency, sleep duration, habitual sleep efficiency, sleep disturbances, daytime dysfunction, and PSQI global score. They found no significant adverse changes in vital signs or blood tests.

Bausell’s Checklist: Fail!

In his book Snake Oil Science, Bausell offers a simple 4-item checklist of things to look for to judge whether a study is likely to be credible:

  • Randomized with a credible control group
  • At least 50 subjects per group
  • Dropout rate 25% or less
  • Published in a high-quality, prestigious, peer-reviewed journal

There were no dropouts in this small study, but otherwise it fails.

Discussion Section

They hadn’t done a credible controlled study to prove that it did work, but that didn’t stop them from trying to explain how it worked. They speculated that the results were due mainly to the velvet bean component. Their rationale? Velvet bean seeds contain L-DOPA, L-DOPA has been shown to stimulate growth hormone (GH), and GH is “closely related to slow wave sleep.” Kind of convoluted, don’t you think? They cite another study that reported a significant increase in GH output in the 2 hours following ingestion of the supplement; but in the present study GH was not measured. Anyway, I don’t think there’s any evidence that patients sleep better when they take pharmacologic doses of GH for medical indications. They also speculated that the Chlorophytum component might have contributed to better sleep because of its saponin content and purported aphrodisiac qualities. The latter is a titillating conjecture, but aren’t aphrodisiacs supposed to excite people instead of putting them to sleep?

They knew they had done a flawed study. They said,

While these findings may be of interest, additional placebo-controlled studies are needed to corroborate these data, possibly including a larger sample size, increased length of treatment with the supplement, and additional time points of measurement. Additionally, further study with each ingredient independently would allow for an understanding of the influence of each on sleep quality.

Grammar Police Grade: B Minus

It always bothers me when a published study contains language errors that should have been noticed and corrected by editors, peer-reviewers, or proofreaders. It doesn’t necessarily mean that the study itself is a poor one, but it smacks of carelessness; and carelessness in language is often associated with carelessness in scientific method and reasoning. This article had a number of things I would have red-penciled, for instance  “The PSQI is self-rated questionnaire” and calling blood tests “bloodborne variables.” Even the title is defective: the borivilianum in Chlorophytum borivilianum should not have been capitalized. Nitpicky, perhaps; but I never see errors like that in prestigious journals like The New England Journal of Medicine.

Other Considerations

I wondered about a couple of points that they failed to address. If the supplement only works indirectly by raising GH levels, wouldn’t it be more efficient to administer GH itself? Would that work? Even if it did, isn’t raising GH levels worrisome? GH is not innocuous: it is a hormone with multiple effects on human physiology.  Would it be wise to raise it just to treat a non-serious condition like mild insomnia? And if velvet beans contain L-DOPA, what are the risks of treating otherwise healthy insomniacs with this powerful anti-Parkinson’s drug? Does it act as a sleeping pill when Parkinson’s patients take it in much higher doses? I don’t think so; in fact, one of the listed side effects of L-DOPA is insomnia.

Why Didn’t They Do a Better Study?

They had funding, they are trained scientists, they even recognized and explained some of the limitations of their study.  Why didn’t they do a better study? Why didn’t they at least use a placebo control? Why did they confuse the issue by testing a product with two ingredients? Why so few subjects? Couldn’t they have eliminated some of those blood tests and used the money they saved to enroll more subjects? Why didn’t they use some objective measurement of sleep quality? A sleep lab might have been impractical and expensive, but a simple device like the Zeo might have provided more objective information than a sleep questionnaire.   Self-reporting about sleep is notoriously unreliable (your grandfather’s complaint that he “didn’t sleep a wink” is belied by the loud snoring you overheard). Their results might seem to support the hypothesis that the supplement is effective, but they are just as compatible with the hypothesis that any intervention will result in self-reported improvements in sleep due to suggestion and expectation. They may have simply provided yet one more demonstration of the placebo effect.

Why do people do studies like this? Why do editors publish them? Why aren’t peer reviewers more critical? When you consider the limited funds available for research, this kind of thing is a real tragedy. Why not throw out 10 or 20 poor-quality research proposals and use the funds for one well-designed study whose results might actually mean something?

Posted in: Clinical Trials, Herbs & Supplements

Leave a Comment (24) ↓

24 thoughts on “Why Do They Do Studies Like This?

  1. Exilapotekare says:

    Sometimes it is as simple as looking at the publisher. The compilation of dodgy OA publishers by Jeffrey Beall (http://metadata.posterous.com/83235355) and the more generic “non-recommended periodicals” at Quackwatch (http://www.quackwatch.org/04ConsumerEducation/nonrecperiodicals.html) are good starting points. As long as you acknowledge that these are subjective lists (and the Quackwatch one is in need of up-dating) and that you don’t necessarily agree with the conclusion at least…

  2. mousethatroared says:

    Why do they do studies like this? I don’t know where to look for it, but I’m curious who paid for the study? Maybe that would give us an idea why they did a study like this.

  3. cervantes says:

    Alas, I’ve been given a few papers to review that were just as bad. One of the worst assigned “life coaches” to people rehabbing from strokes. After a couple of months there was a marginal improvement in their ADL and QOL scores. So coaching works! There was no control. I pointed out that since they were otherwise getting all the usual rehabilitation services, and people regain functioning after strokes naturally, we would hope to see some improvement over time regardless of the life coaches, and this study demonstrated absolutely nothing.

    The investigators were doctors at a highly reputable institution and they had funding for the study. Of course, the important difference here is that I recommended rejection and that’s what happened.

  4. nybgrus says:

    They hadn’t done a credible controlled study to prove that it did work, but that didn’t stop them from trying to explain how it worked.

    First off this is a prime example of how CAM wants to have it any way it can. If there is no possible putative mechanism (reiki, homeopathy) then they cry out that there are plenty of drugs we use that actually work, but we don’t know the mechanism for so that should be analagous. If there is a potential mechanism then they love to try and make one up since they – on some level – realize that aping what actual scientists do is superficially better.

    In any event, I have some contributions to the actual content.

    They speculated that the results were due mainly to the velvet bean component. Their rationale? Velvet bean seeds contain L-DOPA, L-DOPA has been shown to stimulate growth hormone (GH), and GH is “closely related to slow wave sleep.” Kind of convoluted, don’t you think?

    Convoluted indeed. And the “close relationship” is that GH surges are predicated on sleep. In pretty much all data to date, the relationship shown and studied has been sleep’s effect on GH release, not GH on sleep. So already their proposed mechanism is on shaky ground.

    However, there was a 2010 article in The Journal of Clinical Endrocinology and Metabolism (full text) that discusses this issue. They studied people with chronic growth hormone deficiency (GHD) and their sleep patterns. First off, they found poor sleep using the same questionnaire as this herbal supplement study. However, two interesting points. The first is that they found that those with primary pituitary GHD had an excess in intensity and duration of slow wave sleep (SWS) which lead to worse outcomes in sleep quality and quality of life. They also posit that it is the growth hormone releasing hormone (GHRH) levels that is the primary driver of SWS and quality of sleep based upon the varied etiologies and sleep patterns of their GHD study group.

    The best part though (in relation to this article)? Part of the conclusion (emphasis mine) and to answer Dr. Hall’s question (If the supplement only works indirectly by raising GH levels, wouldn’t it be more efficient to administer GH itself? Would that work? ):

    Nearly all GHD patients in our study received chronic replacement therapy (including hydrocortisone/cortisone acetate, l-thyroxine, and sex steroids) to correct associated pituitary hormonal deficits. Although an impact of hormonal deficits and their treatment on sleep quality cannot be excluded, the fact that younger and older patients were similarly treated and sleep was nonetheless more disturbed in older patients does not support a primary role of replacement therapy in the sleep disorders.

    In other words, exogenous correction of known GHD did not improve sleep quality or QoL. So proposing an indirect mechanism of stimulating the production of GH as the putative mechanism for improved sleep quality (while still not out of the realm of possibility) is certainly contrary to the current general understanding and evidence regarding sleep and GH.

    Why didn’t they do a better study?

    You mention spelling, grammar, and a list of obvious limitations to the study design. How about we tack on a terrible literature review as yet another facet of this bad study? This was published in June 2012. The article I cite above was March 2010. My rapid lit review turned it up instantly. How was it not part of their references cited page? They covered all sorts of other topics – other studies on their herbal products, pharmacological studies on sleep (actually an inordinate amount considering their topic), general topics of sleep disorders and the impact on health, reviews of other CAMs and placebos on sleep, toxicity studies of similar herbal compounds, and plenty of studies on L-DOPA and its effect on pituitary hormone release.

    A decent lit review should be a minimum for an article, especially when literature addressing a specific and explicit mechanism exists.

  5. nybgrus says:

    @mouse:

    From the article:

    ZWB, CGMcC, RJB and RJA’s institutions have received a grant from USP Labs. RJB has consulted for Bergstrom Nutrition, OmniActive Health, CEBio, Sigma-tau HealthScience, and Purity Products and his institution has received grants from USP Labs, Kaneka Nutrients, Miami Research Associates, Sigma-tau HealthScience, Mannatech, Advanced Oral Technologies, Purity Products, Life Extension Clinical Research, and Danisco, and he has received speaking fees from Bergstrom Nutrition, royalties from Formulife and manuscript preparation fees from Miami Research Associates.

    All of those companies are natural supplements companies. From my quick search none of them were currently offering either supplement as a product for sale. However, many other companies do and usually the indication is anything but sleep – arthritis relief, sexual performance, etc.

    Interestingly, Miami Research Associates – which paid for the manuscript submission – boasts a full sleep lab at their facility along with 40 CITI certified researchers and in house study capability. Makes me double wonder about Dr. Hall’s question as to why they didn’t use a sleep lab.

    Overall though it look to me like every single company that chipped in support for this study stands to potentially benefit by expanding their line of supplements offered.

    Hardly a smoking gun, but certainly suspicious, I reckon.

  6. They do studies like that so they can advertise products with the claim that “they have been the subject of a study published in a peer-reviewed journal.” A similar misuse of published research I recently wrote about reveals an herb company regularly referring to “double-blind research” and multiple studies without ever directly referencing an abstract. When I dug up the research, one uncontrolled study reports results “approaching or reaching statistical significance.” I compared this to a child telling a parent his grades were “approaching or reaching passing.” A decently done double-blind study found a Chinese herb mix for HIV/AIDS had no benefits over placebo, but created significant gastrointestinal disturbance. Still, it is marketed as being specifically designed for HIV with “strongly antiviral” herbs and claims it can help digestion problems due to side effects of standard antiretrovirals. Of course, the marketing material also says the herb mix has been researched in double-blind studies.

    When the target audience has poor scientific literacy and lacks the curiosity to find and read the actual research, the quality and findings of a study are not nearly as important to marketing as the *existence of* a published study.

  7. Carpus says:

    I’ve reviewed papers like this too for similar journals. I recall one comparing two ‘drugs’ (actually ‘natural’ products like that described here): one the drug under study, the other one with very unclear efficacy in the disease. No placebo. The authors wanted to show that the two were equivalent. The methods they used were wrong – they used a standard superiority trial which cannot prove that two drugs are equivalent. So they demonstrated the two ‘drugs’ were ‘equivalent’ – even though one might have been a placebo – and concluded that the studied drug was ‘effective’ because it worked ‘as well’ as the comparison drug. I recommend that the paper be rejected outright. The authors responded that lots of other similar studies have been published (like that matters). Then the editor wrote me claiming something along the lines of ‘we try to publish as much research as possible and unless there are obvious flaws we publish them. Since other studies like this have been published it must be OK. But we’ll publish your comments along side the paper as well so it’ll be OK.’ Of course, it’s not OK. Just because other bad science has been published, it’s not a reason to publish more bad science.

    Conclusion? The journal is much more interested in the money they get from the author to pay for publication than whether there is any scientific ‘truth’ to the paper.

  8. mousethatroared says:

    nybgrus, thanks!

    yeah, I kinda suspected that the funding came from supplement companies. I also suspect that the study was being done, as skeptical_accupunturist* said, so that advertisers could say “proven to improve sleep in a clinical study!” on the bottle, rather than because someone REALLY wanted to know if velvet bean helps with sleep.

    But I must say, “Velvet Bean” is a vegtable name that begs to be on the produce shelves of a boutique grocery store. Much better than green, wax or long beans. Chlorophytum Borivilianum, not so much, but it does have a certain bio-medical ring to it.

    Gosh, I wonder if these herbs and vitamins are ever marketed just on the basis of sounding like they should be good for a particular ailment…Nahh, I being cynical now.

    *there’s a screen name that must have a story behind it.

  9. stanmrak says:

    silly question – the answer is obvious… follow the money.

    Of course, this is small potatoes. Don’t trust any study:

    http://www.courthousenews.com/2012/06/27/47851.htm

  10. For any new readers, be sure to read Stan Mrak’s profile here:

    http://www.selfgrowth.com/experts/stan_mrak

    Note that he lists Joe Mercola (quack) and Natural News as his “mentors”, and that he runs a website here:

    http://www.antioxidants-for-health-and-longevity.com/

    (ie, don’t bother wasting your time. He’s a coherent rusitchealthy.)

    But as Stan says, just follow the money. On his spammy website he tries to re-sale web-hosting, he brags about his page being in the “top 1% of all internet websites. If you click on the Alexa link above it, you’ll see his site gets almost zero traffic. I guess if he compares himself to every defunct pages on Geocities he’s doing rather well. He also links to a number of websites selling useless and likely dangerous supplements with a referral code for himself. Finally he links to a bunch of books on Amazon that I’m sure he gets a referral fee from too. So basically this guy, who says “just follow the money!” runs a spammy website that spams false information about herbs and supplements and tries to get you to buy something (anything!) so that he can make a buck or two off a referral. Finally on his “About” page we see that Stan has no education or credentials worth mentioning. Stan: get a real job.

  11. Chris says:

    Stan, you are posting off topic. And do where does that legal article explain the difference between Jeryl Lynn, Rubini, Urabe, and Leningrad-Zagreb. And if you do not know what those name signify, then you obviously do not have a clue about the subject.

  12. WilliamLawrenceUtridge says:

    Stan, your response to the existence of some flawed studies and the existence of conflicts of interest in science is to reject all science? Shall we return to the days of epidemic polio, measles, mumps, rubella, death due to infectious disease, general misery and an average life expectancy of 40 years? Or should we perhaps just trust only the studies approved by Joe Mercola? Because Joe Mercola doesn’t sell anything, doesn’t make any money, lives on sunshine and happy thoughts, and certainly doesn’t live in a multi-million dollar mansion paid for by the enormous number of unproven, oversold supplements sold on his website.

    I do agree, conflict of interest is a problem in science, medicine, and in particular, marketing.

  13. Will Shepherd says:

    As an undergrad studying biology and chemistry, the scientific report papers we have to turn in are typically thrown out if > 3-5 grammatical errors exist, i.e., if there are careless errors , the paper isn’t worth grading and you receive a zero. I couldn’t imagine submitting a paper to a journal without thoroughly reviewing it for common errors, much less without a decent trial design. This paper would be a good example to show freshmen “how NOT to write a scientific paper”. Just because it looks and sounds like good science, doesn’t mean it is good science.

  14. Puddle Jumper says:

    In relation to this post about bad journal articles I have a request for information. I am interested in references (books, journal articles, websites) that describe what is good (and bad) science, how to write good scientific papers, and how to understand/dissect/interpret science information.

    Unfortunately, during my undergrad learning the results of science was the primary goal with little focus on how to do and understand good and bad scientific research (which was partially my fault since I was a little unfocused at the time). That is not to say I haven’t learned the basic ins-and-outs of science since graduation but I want more. I’m requesting this information now since I will be entering graduate school soon and will have limited time to learn more about conducting good scientific research, what makes a good scientific study, and reading, understanding, and interpreting scientific journals, books, blogs, etc.

    Thank you in advance.

  15. niftyblogger says:

    Puddle Jumper – congrats on the grad school admission! whoohoo!

    A lot of that stuff is learned along the way. My adviser has been really good about guiding me through how to read a research paper by forcing me to follow journals and present papers. That is what my blog is for..although in its infancy right now. Anyway, advisers are good for that kind of thing. She’s also been pretty helpful in helping to guide me toward designing good experiments, making sure you have all of the controls you need to avoid ambiguity as much as possible. It’s kind of a crash course doing it this way, and it can be frustrating. This, and the For Dummies series, have been my two methods.

    You might want to check out the Cell article “The Importance of Stupidity in Scientific Research.” It’s a short essay that you might find helpful and interesting.

    Simply typing in “How to Read A Research paper” into google yields a bunch of interesting results.
    “https://www.google.com/search?sugexp=chrome,mod=11&sourceid=chrome&ie=UTF-8&q=how+to+read+a+research+paper”

    Hope that helps…I think other people here might have more ideas, though. I’ve only been doing this science thing for about a year now.

  16. niftyblogger says:

    Puddle Jumper – congrats on the grad school admission! whoohoo!

    A lot of that stuff is learned along the way. My adviser has been really good about guiding me through how to read a research paper by forcing me to follow journals and present papers. That is what my blog is for..although in its infancy right now. Anyway, advisers are good for that kind of thing. She’s also been pretty helpful in helping to guide me toward designing good experiments, making sure you have all of the controls you need to avoid ambiguity as much as possible. It’s kind of a crash course doing it this way, and it can be frustrating. This, and the For Dummies series, have been my two methods.

    You might want to check out the Cell article “The Importance of Stupidity in Scientific Research.” It’s a short essay that you might find helpful and interesting.

    Simply typing in “How to Read A Research paper” into google yields a bunch of interesting results.
    “https://www.google.com/search?sugexp=chrome,mod=11&sourceid=chrome&ie=UTF-8&q=how+to+read+a+research+paper”

    Hope that helps…I think other people here might have more ideas, though. I’ve only been doing this science thing for about a year now.

    *EDIT* – Also you could take a statistics class. Having a grasp of how statistics are used can help you see more easily how they are misused.

  17. Chris says:

    Puddle Jumper, make sure you read the book that Dr. Hall referenced under the heading “Bausell’s Checklist: Fail!” She reviewed that book for her first article on this blog. As a coincidence, I read it at the same time I was taking a statistics class.

  18. nybgrus says:

    @puddle:

    The first step in doing and understanding good science is realizing that there is something to understand. So good on ya!

    I would recommend that you start by learning why bad science is bad. Ben Goldacre has a good book as a primer on that. Pick a topic that you find interesting or are otherwise passionate about and see if there is some bad science out there (climate change, evolution, CAM, whatever… mine was evolution because I did my undergrad in it. Bad science in evolution also has the advantage of being really bad which makes it easier to get started. You can advance to Shapiro and adaptive evolution as that is much tougher to explain why it is bad [I have a few comments on it either here or at NeuroLogica) and then combat it. The internet is an endless source of people making bad claims to sharpen your teeth on. Practice, practice, practice.

    Also start learning the common logical fallacies and cognitive biases people fall prey to. Understanding how and why the human mind is so fallible helps spot those mistakes and avoid them yourself.

    You can also try and think about how you might do a study differently and why.

    Some good books to consider:

    Trick or Treatment by Simon Singh
    Bad Science by Ben Goldacre
    Snake Oil Science by R. Barker Baussell
    The Drunkards Walk by Leonard Mlodinow
    Made to Schtick by Chip Heath

    I would also recommend going on YouTube and finding a user by the name of c0nc0rdance (spelled with zero’s not O’s) and check out his videos. He does a lot of them on scientific claims and does a literature review for the video and explains what it all means and why. He also covers logical fallacies and biases.

    And lastly, a sometimes poster (but always welcome voice around these parts) by the ‘nym Prometheus has an excellent two-part post called Anatomy of a Study: A Dissection Guide (Part 1)

  19. nybgrus says:

    oops. link didn’t post:

    http://photoninthedarkness.com/?p=228

  20. fledarmus1 says:

    Puddle Jumper, let me add my congratulations as well.

    Let me also add the advice to search this blog for any posts by Rustichealthy and to read them and the followups to them very carefully. You will find textbook examples of most of the cognitive biases supporting bad science and an array of responses clarifying both the science and those cognitive biases in manners ranging from dispassionate to downright offensive. I have learned an enormous amount about why intelligent, articulate, rational people accept and defend bad science and absolutely refuse to countenance good science based on these discussions.

  21. WilliamLawrenceUtridge says:

    It’s a more general reference, but I would also recommend Mistakes Were Made (But Not by Me) by Carol Tavris and Elliot Aronson, which does a good job of emphasizing how bad we are at reasoning but how good we are at justifying ourselves. A quick, punchy book.

    If you’re looking into creationism criticisms, I would recommend Thunderf00t’s (zeros as well) Why Do People Laugh At Creationists? series on YouTube. Effective use of simple math and facts to demonstrate how stupid creationism is.

    These are more skeptical sources than scientific criticisms though. But I very much agree with nybgrus’ statement that this sort of nonsense is excellent for general training in scientific critical thinking and what science is, criticisms of creationism in particular place heavy emphasis on defining science and explaining why creationism isn’t it (see the Edwards v. Aguillard amicus curiae by 72 Nobel laureates hosted on Talk.Origins for a neat overview of science-versus-not). Talk.Origins has an enormous amount of information and is generally a fun, slightly bitchy read.

  22. DavidRLogan says:

    Fun read and discussion. Thanks for the comments and links, everyone.

  23. DW says:

    I just wanted to point out that it’s a mistake to assume that “open access” means the journal is no good. Jeffrey Beall’s list of “predatory” open access publishers is a very good resource. But we should not assume that the only reason researchers would pay to have a study published is that their results are not otherwise publishable. Open access is a legitimate publishing model, and many of the top tier peer-reviewed journals offer an open access option.

  24. nybgrus says:

    @puddlejumper:

    go to this thread:

    http://www.sciencebasedmedicine.org/index.php/quackademic-medicine-trickles-out-to-community/comment-page-1/#comment-95850

    and pick up what herbal is laying down and see if you can desconstruct why (s)he is so wrong. ‘Twouldn’t be a bad start for you. :-D

Comments are closed.