Articles

Woo-omics

Every so often, I come across studies that leave me scratching my head. Sometimes, these studies are legitimate scientific studies that have huge flaws or come from an assumption that is very off-base. Other times, they involve what Harriet Hall has termed “tooth fairy science,” wherein the tools of science are used to study a phenomenon that is fantastical, whose very existence hasn’t been demonstrated. Many such studies, not surprisingly, are studies of “complementary and alternative medicine” (CAM) or “integrative medicine” (IM). Modalities like reiki (which is faith healing that substitutes Eastern mysticism for Christian beliefs) and homeopathy (which is, when you boil it down to its essence, sympathetic magic) fall into the category of therapeutic modalities that are based on fantasy but are studied as with the latest tools of science, producing no end to confusing noise. This “tooth fairy science” has, over the last few years, reached its epitome in the application of the latest genomics technology to, in essence, magic, and I’ve recently come across an incredible example of just such a thing. But, first, let’s take a step back to what is going on in medical science now before I introduce a concept that I’ve dubbed “woo-omics.”

A prelude to woo-omics: Genomics, proteomics, everywhere an “omics”

One of the most difficult problems in science-based medicine is how to do a better job identifying which patients will respond to which treatments. Clinical trials, by their very design, have to look at average responses in populations. In essence, a treatment is compared to either placebo or standard-of-care, a choice mainly driven by ethics and whether effective treatments exist for the condition being studied. It is then determined using statistics whether a significant difference exists between the two groups. The difficulty, as any clinician knows, is applying the results of clinical trials to individual patients. In any population, there is, after all, a range of responses to any drug or treatment, and it would be desirable to be able to predict which patients will fall at the end of the bell-shaped curve where the treatment is most effective and which will fall at the end of the curve where the treatment works poorly or not at all.

In the past, predictors of response (or, just as importantly, lack of response) to any given therapy were crude. For instance, take my specialty of breast cancer. If the tumor does or does not express the estrogen receptor (ER) determines whether it will be sensitive to antiestrogen therapy. It also can to some extent predict whether a tumor will be responsive to chemotherapy, as ER(+) tumors tend to be less responsive to chemotherapy than ER(-) cancers. However, that’s not entirely true in that there are subsets of ER(+) tumors that are responsive to chemotherapy. It’s just that, until recently, we couldn’t identify them other than by administering adjuvant chemotherapy and observing the response. Fortunately, in the last several years, clinicians have started to get some help in this area. For example, a 21-gene expression assay known as the Oncotype DX assay can determine whether an ER(+) tumor is likely to be responsive to chemotherapy or not. Basically, the test looks at a panel of genes and produces what is known as a “recurrence score.” Low recurrence scores mean a lower likelihood of recurrence and less benefit from chemotherapy, while high recurrence scores indicate a higher probability of recurrence and a greater benefit from chemotherapy. As a result, the primary tumors from most women with ER(+) cancer that has not as yet metastasized to the axillary lymph nodes are tested with Oncotype DX, and patients whose tumors have low scores are not offered chemotherapy, thus thwarting what Mike Adams calls the “breast cancer industry” and its relentless search for profits by pumping women with breast cancer with expensive and toxic drugs. Oh, wait, it was scientists and doctors who came up with the Oncotype DX test. Damn, we’re good at working against our own interests and those of our pharma masters.

Be that as it may, these days, the search for predictors of response, prognosis, and therapies most likely to do good has moved into the realm of what we now call “omics.” The term “omics” as it is used today originally came from genomics, which is, put very simply, the study of the entire genome (i.e., all the genes in an organism). It then expanded to be used for proteomics, which, again put very simply, is the study of all the proteins expressed by a cell type, organ, or organism. Since then, the term has metastasized to many, many areas of biology, such as metabolomics, secretomics, lipidomics, and many, many others. Here’s a general schema of what I’m talking about:

The problem with all these “omics” is that they are hideously complicated, with interactions of thousands of genes, proteins, and other entities that must be made sense of in order to understand what is going on. Indeed, arguably the reason we never bothered with these sorts of analyses before is that, until the last 10-20 years quite simply they were impossible. The computing power and algorithms necessary to do them simply didn’t exist and had to be developed. Neither did the technology. Then, beginning in the late 1990s, techniques were developed to measure expression profiles that included every known gene in the human genome. Building on techniques developed for the Human Genome Project and other genomics initiatives, in the early 2000s, we had cDNA microarrays, the ability to scan thousands of single nucleotide polymorphisms (SNPs) and look for associations with diseases, and the like. Systems biology started to come into its own, wherein biology was studied not so much at the level of the single gene and protein but by looking at expression and activity data and constructing networks that look like this:

The result of the new systems biology and “omics” has been a torrential flood of data that’s far ahead of our ability to analyze it fully. As the cost of sequencing a genome has fallen from hundreds of thousands of dollars to less than $10,000 (soon to be less than $1,000), genome sequencing will soon fall to within the price range of other commonly used medical tests. (CT scans and MRIs cost around $2,000 or so, and the Oncotype DX test, for example, costs around $3,000.)

Unfortunately, even as the flood of data accelerates, successful strategies for actually using that data clinically have been elusive. Indeed, last year, around the time of the tenth anniversary of the completion of the Human Genome Project, there were a series of articles asking, basically, “Where are all the cures we were promised?” Of course, as I’ve pointed out before, the sequencing of the human genome (and now all these other genomes, as is being done in the Cancer Genome Atlas, for example) has been the easy part. The hard part is making sense of it all and relating differences in individual genomes to specific diseases and to the discovery and validation of biomarkers for response to specific therapies. Just looking at one example can demonstrate why it’s so hard to make sense of this data and to figure out how to use it to develop cures to diseases like prostate cancer. Does all of this mean that all the information we’ve gathered and connections we’ve made so far in the Human Genome Project, the Cancer Genome Atlas, and other similar projects that have tried to relate genomics data to human disease, prognosis of disease, and response to therapies useless? Of course not. It’s just that the speed with which this data will result in real cures was arguably oversold. Right now, the situation is confused and uncertain. and we are still very far from the vision of truly personalized medicine that so many see “omics” as the path towards.

Which makes it perfect for purveyors of unscientific medicine to leap in and take advantage of. First, I’ll introduce the concept of “woo-omics” (which, to my knowledge, is a term coined by me to refer to the application of omics principles to woo). Then I’ll show you a truly ridiculous example, after which I’ll try to bring it all together at the end.

Woo-omics

Promoters of “complementary and alternative medicine” (CAM) and “integrative medicine” (IM) realize that, if CAM/IM is ever to be taken truly seriously they need to provide a patina of scientific respectability to it. Whether it deserves that respectability or not is secondary to the endeavor of promoting unscientific medicine as being co-equal with science-based medicine (SBM). Also, promoters of CAM/IM are nothing if not mimics. They’re very good at jumping on the latest scientific bandwagon as a means of trying to achieve this patina of scientific respectability. Not surprisingly, believers in CAM/IM, aided and abetted by the National Center for Complementary and Alternative Medicine (NCCAM) have jumped on the “omics” bandwagon in general and the genomics bandwagon in particular. I first became aware of this four years ago, when I noticed an RFA from NCCAM for omics studies in woo entitled Omics and Variable Responses to CAM: Secondary Analysis of CAM Clinical Trials:

This initiative will be used to stimulate omics analysis by CAM investigators. Multiple NCCAM and other NIH CAM clinical studies have appropriately banked patient samples from several sources including a variety of tissues. Therefore, this initiative will encourage use of these already acquired samples from well-designed clinical studies for identification of genomic, proteomic and metabolomic and other omic variants that may be markers or classifiers for the level of response to CAM interventions. Trials need not exhibit significant differences between arms. This analysis may be applied to studies with negative results as a function of high patient variability in the treatment arm, which may represent responder/nonresponder phenotypes. Since this work would use samples already collected, the participants may not be consented for genetic studies and would have to be reconsented and samples de-identified. Guidelines for data access and release can be found at http://www.fnih.org/GAIN/GAIN_home.shtml.

“Well-designed clinical studies”? You keep using that term. I do not think it means what you think it means. Of course, any project proposed for this RFA would by its very nature have to be a massive fishing expedition. As much as I like the new technology that can produce such copious amounts of data, I’m not a big fan of such studies even in conventional medical trials, where the treatment studied has clear efficacy, a molecular mechanism of action, and thus a reason to think that there might be “omics” profiles that correlated with magnitude of response. For one thing, such studies are not hypothesis-driven, and investigators will not have much of an idea of what they might expect to find; that is, if they find anything at all. Thus, in the absence of clear criteria for what constitutes a significant finding, there is a significant risk of all sorts of spurious findings being labeled as “significant.” Of course, if researchers look for a large enough number of correlations, they will inevitably find them even if they are not true indicators of causation just on the basis of random chance alone. Such analyses are in essence post hoc exercises; it is far better to build such analyses into a study prospectively from the very beginning. The data is cleaner that way, and the statistical analysis has more power.

That being said, I will concede that for some clinical trials, post hoc analysis of the tissue specimens can be very useful as a means of generating hypotheses. Specifically, if there are clear-cut “responders” and “nonresponders,” subjecting samples from these patients to “omics” analyses can provide clues to help investigators determine why some subjects respond and some do not. Useful hypotheses to test can be generated from such correlations. In fact, this is the very reason why I found the NCCAM proposal so pointless: precious few of the modalities tested in NCCAM-funded grants have ever been shown to have even a glimmer of efficacy greater than that of a placebo, and when you start talking about reiki or other “energy healing” techniques, homeopathy, or various other techniques with no clear basis in science or reality, you’re talking about magic. Given that, it’s a waste of resources, specifically precious grant money that could go to fund far more worthy endeavors. While this sort of approach might be somewhat useful for various herbal remedies, mainly because such remedies are drugs (impure drugs with many components, but drugs nonetheless), they would be utterly useless for practically every other CAM modality. For example, does anyone imagine that genomics, proteomics, or any other “omics” would identify “responders” and “nonresponders” when it comes to Reiki therapy. I don’t think so–unless there is an “omics” profile for credulity or the placebo effect. Or maybe they will discover a genomic or proteomic profile for credulousness or personal belief structure.

Fortunately, with the hindsight of four years, I’ve only been able to find two projects funded under this mechanism. Unfortunately, they are grants entitled Omics and variable responses to placebo and acupuncture in irritable bowel syndrome and Dietary calcium and magnesium, genetics, and colorectal adenoma. The former project is based at Harvard, specifically at Beth Israel Deaconness Hospital, and refers to trying to determine biomarkers of response to acupuncture therapy in patients with irritable bowel syndrome using -omics technology applied to blood samples:

Irritable bowel syndrome (IBS) is a chronic medical condition which affects 10-15% of the population in industrialized countries. IBS is associated with a profound impairment of patient’s quality of life, significant loss of productivity, and $2 billion in direct health care costs in the US alone, mainly due to unsatisfactory treatment. Patient heterogeneity in terms of disease pathogenesis, symptom expression and psychological co-morbidities are among the factors implicated in the variation of therapeutic responses. Therefore, the need to develop biological markers that can distinguish patients likely to benefit most from specific forms of treatment, and as importantly, from a placebo treatment, is imperative. We have recently concluded a 6 week randomized clinical trial in IBS patients (n=262) to examine the effects of placebo and acupuncture treatments, compared to a natural history control group. Blood samples were collected at baseline and at 3 and 6 weeks of follow-up. The positive results of this trial and sample availability allow us to perform a secondary Omics analysis examining patient variability in response to treatment. Our primary hypothesis is that serum proteins participate directly or indirectly to the brain-gut interactions associated with clinical responses in IBS. Our second hypothesis is that patients who clinically respond to placebo or acupuncture treatment have distinct serum protein profiles from those who don’t respond. A third hypothesis is that placebo or acupuncture treatment results in changes in the serum proteome. We will incorporate various high throughput experimental approaches, some of them targeted to specific biological pathways and others leading to unbiased discoveries, to pursue the goals of this study. The serum proteomics analysis will be complemented with an investigation of common polymorphisms in genes of interest that emerge. Findings stemmed from such studies could lead to improvement of inclusion/exclusion criteria in randomized clinical trials based on biological screens. In terms of personalized medicine, findings could result in methodologies to enhance medication responses, adjust the doses or select medications with fewer side effects.

The study used as the basis in preliminary data for the above grant appears to be Components of placebo effect: randomised controlled trial in patients with irritable bowel syndrome. Guess who the first author was before clicking on the link. Go on, guess. OK, I’ll tell you: Ted Kaptchuk. And what was the finding of the study? Basically, it looked at IBS patients treated either with sham acupuncture or acupuncture plus “augmented interaction” with the practitioner. In the former group, acupuncturists spoke little with the patients and interacted with them as little as possible, while in the latter group they spent a lot of time with the patients asking them about their lifestyles and various other issues. Basically, both of these groups did better than the observation-only control group, while the “augmented interaction” group demonstrated more improvement in symptoms. It was actually not a bad study, although all it showed was that sham acupuncture works as a placebo and that a closer interaction with health care practitioners enhances the placebo response, which is nothing new.

As for all the specimens, given that the placebo acupuncture group and the “augmented interaction” groups were re-randomized half way through the trial, it’s hard to see how applying omics technology to these samples is likely to provide much useful information. Moreover, the design of the trial incorporated a nested substudy of true acupuncture that allowed the investigators to tell subjects truthfully that, if they were randomized to an acupuncture group, they had a 50% chance of receiving “true” acupuncture. The results of this sub-study were ultimately reported in a psychiatry journal and in a followup article in the American Journal of Gastroenterology. Interestingly, the latter report says in essence nothing about whether true acupuncture produced better results than sham acupuncture. Indeed, Kaptchuk goes out of his way to avoid discussing that question in the article and even lumped together acupuncture and sham acupuncture groups when assessing “responders” and “non-responders,” which strongly suggests to me that there was no difference between the sham and “verum” acupuncture groups.

And Dr. Efi Kokkotou, along with Ted Kaptchuk, propose to do omics studies on blood specimens from these subjects, even though there was no real effect due to the intervention and even though the intervention group was re-randomized at three weeks. I’m not sure what Kokkotou and Kaptchuk think they’re going to find, but with multiple comparisons I’m sure they’ll find something and that that something is very likely to be noise or spurious results.

All to the tune of $521,646 in FY2011 and a total of $1,514,997 thus far, your tax dollars hard at work again, courtesy of NCCAM. Interestingly, listed as one of the publications resulting from this particular grant is a study I discussed earlier this year in which Kaptchuk and company tried to claim that they could harness placebo effects without deception. As much as they tried to spin it otherwise, they couldn’t. Another paper whose work was funded through this grant claims to have found a serum correlate of placebo response in IBS. One sentence completely undermines this claim:

Since the current study was hypothesis generating, we did not apply Bonferonni/Dunn correction for multiple comparisons to any of our analyses.

Sorry, but being “hypothesis-generating” is not an adequate reason not to correct for multiple comparisons. One wonders if Kokkotou and Kaptchuk will apply the same logic to omics data. If they do, they’ll have at least hundreds of potential “biomarkers.”

While the abuse of omics technology by researchers like Kaptchuk (and, by the way, Dean Ornish as well) is egregious, I had no idea that I hadn’t seen the worst of it.

And now…Ayurvedomics!

I don’t remember how this came to my attention, but it did. It’s an article that appeared in ACS Chemical Biology entitled Ayurgenomics: A New Way of Threading Molecular Variability for Stratified Medicine. I kid you not. It’s courtesy of Mitali Mukerji at the Institute of Genomics & Integrative Biology (IGIB) in New Delhi, India. Her webpage on the IGIB website even lists her interests as the application of polymorphisms in mapping mutations and disease origins in SCAs; Ayur-genomics: exploring the principles of predictive medicine in Ayurveda; and Alu elements in genome organization and function. It all sounds so science-y, and—who knows?—interest numbers one and three sound reasonable enough. But interest number two? Not so much. Get a load of the rationale for the article. After discussing the difficulties in genome-wide association studies in “pinning down human physiology to a few loci,” Mukerji writes:

We found this challenge very stimulating, and our understanding of population-wide variability across Indian populations (IGV Consortium)(6) provided a major thrust to further our quest for understanding the variability in healthy individuals. The head start came from the fact that there exists an exquisitely elaborate system of predictive and personalized medicine in India, i.e., Ayurveda, which has been practiced for over 3500 years. The system already has a built-in framework for stratifying healthy individuals who differ in susceptibility to disease and response to drug and environment. In contrast to the empirical approach of contemporary medicine, the Ayurveda therapeutic regimen is tailored to an individual’s physiology. Though this system has fueled many drug discovery approaches and some attempts into its integration in pharmacogenetics have been made,(7, 8 ) a systematic analysis of underlying principles has been lacking. In order to undertake integration of this most ancient system of medicine, which is scripted in Sanskrit, with the language of modern genomics and medicine, we undertook this endeavor with a trans-disciplinary team of researchers. For the first time we could demonstrate molecular evidence for these concepts and build a framework for “Ayurgenomics”, which can provide impetus to personalized medicine.(9, 10) We provide a perspective of the concepts and also the further prospects in this field (Figure 1). The original Sanskrit verses with their meanings are available as supplementary online material in our prior publications.

Now, it was news to me that Ayurveda had “fueled many drug discovery approaches”; so I looked up the two articles referenced. Like Mukerji’s article, these references looked at the three major constitutions postulated in Ayurveda under the Prakriti classification. One article was in a woo journal of highly dubious value and postulated a “reasonable” correlation between Prakriti classification and certain Prakriti types, while the other article was a commentary suggesting using Prakriti and asserts that the “concept of Prakriti or human constitution plays a central role in understanding health and disease in Ayurveda, which is similar to modern pharmacogenomics.” Basically, the whole article argues that, because a few useful compounds from Ayurveda, that Ayurveda could be useful as a basis for drug discovery and the development of “personalized medicine.”

Not knowing what the heck Prakriti is, I decided to look it up. Thanks to Wikipedia and other sources, I now know that in Hinduism Prakriti is the “basic nature of intelligence by which the universe exists and functions” and the “primal motive force.” In Ayurveda, it’s used to refer to a body type in Ayurveda. Here is a test to determine one’s Prakriti. Based on the results, one is classified according to three Doshas: Vata, Pitta and Kapha. These three Doshas can be combined into ten ways to produce ten body types. Now, certainly some of these characteristics likely have something to do with health (“overweight, difficult to lose weight,” for instance, or “fair skin, sun burns easily,” the latter of which could indicate susceptibility to skin cancer). However, just because body types based on a system of classification that is more or less based on an Indian form of vitalism (“primal motive force” or “basic nature of instelligence”) might coincidentally have something to do with discoveries found later does not mean that the basis of the system upon which these body types are based has any external scientific validity. One might as well perform omics analyses on people to predict responses to homeopathy.

Yet none of this prevents Mukerji from proposing this system for “weaving the threads of molecular variability through Ayurgenomics”:

This is proposed even though Mukerji basically points out that Prakriti is basically very much like the four humors in “Western” medicine and the five elements in traditional Chinese medicine:

Ayurvedic practitioners deconvolute the “mixture impression” thus obtained to identify proportions of Vata, Pitta, and Kapha in an individual’s Prakriti. This “subjective” assessment (which was objectivized to a scoring system through a questionnaire) considers different phenotypic attributes of an individual and links these multiple windows to create intraindividual phenotype-to-phenotype links. A disease according to Ayurveda is a perturbation of Vata, Pitta, and Kapha in an individual from his or her homeostatic state. Ayurvedic treatment aims to bring it back to its native state by appropriate dietary and therapeutic regime.

And:

Ayurveda describes not only the functional attributes of Vata, Pitta, and Kapha but also their contribution on different scales in seven different constitutions. Therefore, Ayurveda already has a stratified approach as its basic tenet for personalizing therapy. Thus we felt that integration of this stratified approach could complement approaches to development of personalized medicine while gaining insights into systems biology. We realized that before making any attempt toward this endeavor we would have to address an ontological challenge in connecting together the literature from Ayurveda, modern medicine, and molecular biology.

Again, combining Ayurvedic literature with modern medicine and molecular biology would be a good thing is not explained. To me, that would be a lot like taking the writings of Hippocrates and combining them with the latest cutting edge molecular and systems biology techniques. It would be highly unlikely to be informative or helpful.

Not surprisingly, Mukerji cites a lot of her own work in this review article, including an article from 2008 entitled Whole genome expression and biochemical correlates of extreme constitutional types defined in Ayurveda, which Steve Salzberg, a systems biologist and skeptic, looked at in 2008. He was not impressed, and neither was I when I read the paper. Basically, Mukerji and her co-investigators partitioned subjects into the three Doshas based on Ayurvedic principles, but in reality the vast majority of the classification is simply based on body type and body habitus. That’s why, when the authors found that genes associated with an elevated risk of cardiovascular disease are more highly expressed in Kapha males, it’s not particularly surprising, given that Deepak Chopra and others characterizes Kapha as prone to obesity. In other words, so what if Mukerji found differences in gene expression based on Ayurvedic body types. They’re different body types, and we already know that people with different body habitus can be prone to different kinds of diseases. Ayurveda adds nothing new to this knowledge, much less suggests a rationale upon which to base a new “omics” discipline. Taking studies like this and proposing something as ridiculous as “Ayurgenomics” does nothing more than show how far believers in pseudoscience will go to try to conjure up a seemingly scientific justification for their woo. Remember, Doshas resemble, more than anything else, the Indian version of the Four Humors. Do we try to fit humoral theory into a new “omics” discipline? No, we don’t, although I fear that some day someone might try.

Unfortunately, as Steve Salzberg points out, NCCAM is funding grants like R21AT001969, which funds the “Ayurvedic Center for Collaborative Research.” A quick search for NCCAM grants funding Ayurveda on NIH Reporter reveals several grants in Ayurvedic medicine funded by NCCAM, including Ayurvedic alternatives in autoimmunity and A whole systems approach to the study of Ayurveda for cancer survivorship. One wonders how long it will be before NCCAM starts funding “Ayurgenomics” projects.

Woo-omics ascendant?

The whole idea of doing omics on tissue samples from NCCAM-funded studies is about as good an example of putting the cart before the horse as I can think of. There is no doubt that individual biological variability is problematic for all medicine and is the main reason that clinical trials of sufficient size and power are needed to separate random biological variation from real treatment effects. However, if a treatment is efficacious in a significant part of the population, it should demonstrate efficacy in a properly designed clinical trial, and if there are significant differences in how people respond sufficient to produce two classes (responders and nonresponders) that should show up as well. Studying omics in the absence of clear evidence that these two conditions apply (clinical efficacy and definite groupings of responders) is about as likely to produce useful information as a 30C homeopathic remedy is to contain a single molecule of the original remedy upon which it’s based. Similarly, trying to yoke genomics to a system of religion-inspired prescientific “medicine” like Ayurveda is no different than proposing to base genomics profiles on the four humors.

The really frustrating aspect of woo-omics is that, in these constrained budgetary times, any money devoted to such studies is money diverted from more scientifically worthy projects. A justification can be made for such retrospective analyses when there is definite evidence of efficacy and definite evidence of classes of responders and nonresponders, as is found in many drug trials, but, barring that, it’s far more likely that good money will be thrown down the pit after bad to find out whether there is an “omic” profile for woo. Worse, it’s already difficult enough to validate candidate biomarkers based on existing omics data. Between measuring the expression of every messenger RNA and microRNA in a cell simultaneously and performing next generation sequencing and various other techniques that permit the sequencing of whole genomes and expression data simultaneously, we can measure more things in a cell than ever before. However, because we can measure so much, it’s all just noise when there is no actual effect detected between two treatments, and it’s a lot of potential noise because thousands of comparisons are being made simultaneously in even a simple comparison. That’s why adding magic to the omic mix will not help, as it will only accentuate the noise. Unfortunately, magic is exactly what defines woo-omics.

Posted in: Basic Science, Diagnostic tests & procedures, Medical Academia, Science and Medicine

Leave a Comment (25) ↓

25 thoughts on “Woo-omics

  1. kathy says:

    The trouble with the funding of woo is that there are no criteria for excluding anything at all. Homeopathy, acupuncture, chiropracty, Ayurveda, shamans and gurus of any and every variety: where do you draw the line?

    In fact they seem to be proud of the absence of any line that says “No!” … “My reality is different from your reality and both are equally valid”.

    It’s a bus with no brakes, on a winding road, and the driver smiles and declares to the nervous passengers that he is happy with the situation … and it seems to me it’s gathering speed too.

  2. DrRobert says:

    The most frustrating thing for me is when a junk study gets slipped into a rag journal, and then the CAM community points to that one study, which is contrary to every other study on earth, as proof of their therapy. I wish the CAM community understood the importance of the systematic review.

    http://www.skepticalhealth.com/2011/11/19/medical-evidence-the-systematic-review/

  3. cervantes says:

    John Ioannidis has written some very important papers about data mining in genomics. People have finally gotten the message, that they should have understood from the beginning, that if you go through a whole lot of data points — in the case of these studies of the association between genetic variants and diseases, we’re talking thousands — you will find spurious correlations. The p value can only be interpreted in light of Bayes theorem. If the prior probability of an association is very small, then it is still highly unlikely, even if your p value is also small. Science is a process of learning — it builds continually on prior evidence. If something doesn’t make sense based on what we already know, it’s unlikely to be the explanation for an observation. (Bayes theorem is extremely important, and in biomedical research, we’ve gotten stuck in a Gaussian world that many of the people who do research, even some prominent investigators, fundamentally do not understand. As Ioannidis demonstrated, most published findings are false.)

    I think this is a good summary of what Dr. G keeps trying to tell people.

  4. thatguybil81 says:

    This is why federal funding is of questionable use. The spending of other peoples money on projects that fullfill a political, idealogical need tend to end up being malinvested.

    Does anyone believe that “big Pharma” would waste money on this type of research?
    Of course not. Only in a federally funded world would research like this get funding.

  5. rork says:

    “and it’s a lot of potential noise because thousands of comparisons are being made simultaneously”.
    That is just the way it is with big data. It’s the same for everybody whether your ideas have plausibility or not. The point is not that there is too much noise. One point is whether the hypothesis is junk from the get go. The other is that, if you want to call it science, you are obliged to estimate, as carefully as you can, what the chances are that you are fooling yourself. How many editors, reviewers, and readers really understand what that means is big data situations? Where would they have learned it? I’m not saying those points aren’t addressed in the article. I’m just doing my additional missionary work.

    So scientists and woo-ists, you do have to sweat the multiple comparisons thing – just like the rest of us. You are not exceptional. If you don’t sweat it in your paper, the reviewers should point that out as a show-stopping flaw. Even papers treating real scientific hypothesis have tried to wiggle out of estimating the chances that conclusions are false positive by saying that no decision is being made – which is always a load of bullshit. If you aren’t making decisions based on the data, there was no point to collect and analyze it, or read about it. Some don’t even try to wiggle out, they just say nothing and rely on the kindness of reviewers in being clueless, which is not a bad bet. Some papers even give methods for worrying about multiple testing, but then fail to give the result, so we know reviewers were in a deep slumber. Editors just don’t care. I’d rather not give specific examples cause it may be hazardous for me.

  6. DugganSC says:

    “the pplication” should be “the application”. Also, you have a sunglassed-smiley face in your section on Ayurgenomics right after “(7, “. I suspect that you have an 8 followed by a right parenthesis and the blog corrected it into an emoticon for you. I get that all of the time on an RPG board I post on.

    As regards the central idea of applying the “omics” to figuring out who to treat, I half expect some people from the other side to chime in saying you’re being hypcritical when half of the articles on this site are about how CAM doesn’t have a consistent diagnosis method such that if you took ten practitioners and showed them the same patient, they’d wind up finding ten different outcomes (and wouldn’t be able to distinguish the healthy patient from the ill one). Here, they’re doing research to try to find who their technique might work on. For practices like chiropractry, which by all appearances works well as long as it’s only applied to lower back pain, this might be the beginning of finding the legitimate kernel of medicine in them. :) Not saying that I’d agree with them that you’re being hypocritical, but I can see it coming.

  7. daedalus2u says:

    There was a really nice talk at Neuroscience 2011 on epigenetics by Andrew P. Feinberg. He started his work on the epigenetics of cancer and now is branching out into the epigenetics of everything. He has developed a gene chip that looks at the methylation patterns of the genome and is finding all kinds of changes, even in the same tissues, across the lifespan.

    I don’t remember all the details, but this seemed to be a source of a lot of the variation in phenotype of highly inbred strains of animals. Even though they are essentially identical genetically, there is still substantial variation. That variation could be explained by epigenetics. The differences in phenotype between different organs is due to epigenetics, not genetics.

    This really calls into question the genome wide association studies. If a particular genome can produce thousands or tens of thousands or hundreds of thousands of different phenotypes, the association of a particular phenotype with a particular genome may be due to epigenetics and not genetics.

  8. Harriet Hall says:

    DugganSC,

    Yes, Dr. Gorski wrote “pplication.” Then you wrote “hypcritical.”
    I can’t imagine that any reader failed to understand what each of you meant.

  9. DrRobert says:

    DugganSC,

    Chiropractic has only been shown to work marginally well on a couple of times of lower back pain. It’s no more efficacious than an ibuprofen or a back massage. Would you rather a 10 cent pill, or hundreds of dollars worth of quackery and dozens of hours of your life wasted?

  10. WilliamLawrenceUtridge says:

    @ thatguybil81

    Federal funding is extremely valuable for things like basic research, as well as systematic reviews or clinical trials independent of pharmaceutical companies because of the reduced bias. A balance of private funding and public (government) controls is normally a good thing, better than either one on its own. While public funds may be distributed spuriously (as what happens with the NCCAM) private industry tends to churn out studies that are unduly positive, even as they may be less susceptible to political pressure. Also, because they are solely motivated by profit, they will happily dump money into medications that are popular rather than effective (i.e. me-too drugs, viagra and the like). There are advantages and disadvantages to both. Come to think of it, the only real advantage to private industry studies is that we don’t have to pay for them (except in terms of the cost of medications – which is grossly inflated because of the need to pay for these studies along with salaries, overhead, advertising and conference trinkets).

  11. David Gorski says:

    Yes, Dr. Gorski wrote “pplication.” Then you wrote “hypcritical.”
    I can’t imagine that any reader failed to understand what each of you meant.

    Yes, spelling/typo flames are a rather low order of criticism; thanks for pointing that out. :-)

  12. rork says:

    daedalus2u: My not-really-expert impression is that the ordinary SNP-measuring GWAS study likely can’t find epigenetic factors – there won’t be a linkage disequilibrium. Some methylation might stay put for generations, but generally it’s a drunken sailor, stumbling into on and then off at will, and retouched at every germ cell to boot. There are exceptions at some screwed-up-imprinting genes, but there’s a DNA mutation associated with the methylation defect. Please teach me better, anyone. I figure you are thinking of methods where you just directly measure the methylation status at many locations. Pyro-sequencing with massively parallel sequencing techniques for example (arrays can work too). I agree it’s exciting. In the old days we tried doing such work with RGLS gels (2D gels of DNA fragments cut with restriction endonucleases that are methylation sensitive, often NotI). It was like having about 1/1000th of an eye compared to the newer techniques. The old days were brutal.

  13. daedalus2u says:

    Rork, the point was that epigenetic programming goes on all the time. There is differential epigenetic programming of somatic cells as well as germ cells and that epigenetic programming changes over the lifespan of the organism. You can’t read out that differential methylation unless you look at the actual tissue compartments under consideration. In the context of neuropsychiatric disorders that is the brain (mostly but other tissue compartments matter too). Those tests would be destructive and the methylation patterns likely change over different regions of the brain and likely change over the lifespan and according to environmental effects.

    He gave an example in honeybees, where queens and workers were not distinguishable via DNA methylation, but the methylation of workers changed over their lifetime. Workers start out as nurses in the hive and progress to collectors of honey and pollen which shows up in methylation patterns. If you remove all of the nurses, some returning workers revert to nurses and some (but not all) of their methylation patterns revert too.

    It was a massively parallel operation, they looked at methylation at 7 million sites. There were about 25 sites that seemed to account for ~90% of the cancers they looked at (~450 different types). The cancer methylation seemed to be mostly in the parts of the genome that identify a cell as part of a specific organ.

    The only difference between somatic cells, liver, brain, kidney, heart is the epigenetics. The genomes of those cells are the same.

  14. BobbyG says:

    I love this site. I linked this in (with excerpts) on my REC blog, http://regionalextensioncenter.blogspot.com. I was blogging on the topic of “personalized medicine,” epigenetics,” etc, when I popped in to SBM (as I do every day) and saw this post.

  15. Angora Rabbit says:

    We’re in the thick of -omics right now, having just finished NGS on several transcriptome sets to look for disease susceptibility markers in a lovely unbiased search. We actually got a small set of very promising candidates, five of which even map to the relevant pathway. But at day’s end this is still just an association (it’s only a model – shh!), and you can bet that we will individually characterize each candidate to verify their expression difference and then test for their actual relevance to our disease endpoint. The description of the CAM study above is horrifying; once can’t handle the data tsunami (how many hard drives have I bought for Gigs of data in this study?) without proper stat analysis.

    thatguybil81: WLU said it right; Fed funding is the right place for this. Paid for mine. And you can bet that the Pharma houses are all doing this, in fact long before we could do it at the Fed level, as their risk taking for early technology will always proceed that from Fed funding (as anyone who has sat in NIH Study Section will affirm). The spouse of one of my former postdocs was hired in the late 90s at Merck to run the genomics on every patient in every drug trial they had. You can bet they wanted to know about the basis for drug response, both at that level of technology and in anticipation of future technologies, like the NGS we now have.

    “The serum proteomics analysis will be complemented with an investigation of common polymorphisms in genes of interest that emerge.” The scientific phrase for this is Fishing. “Genes of interest that emerge”??? Cannot believe this got through a study section.

  16. Angora Rabbit says:

    Dadaelus2U: Epigenetics is so old. :) Check out James Lupski’s work at Baylor on Non-Allelic Homologous Recombination. I heard him at an NIH symposium in Sept and it blew me away.

  17. @DugganSC – Since my comments are incredibly error rich, I usually don’t correct other’s writings. But occasionally, I do see an editing error in an article that is genuinely confusing (I am easily confused). I’ve found that sending an email to the writer (links off their page on this site) will often result in a correction. It’s a good way to avoid getting pinged on for typo alerts.

  18. Bayes theorem is extremely important, and in biomedical research, we’ve gotten stuck in a Gaussian world that many of the people who do research, even some prominent investigators, fundamentally do not understand.

    cervantes, you’re much too polite. MOST prominent biomedical investigators do not understand this.

  19. David Gorski says:

    Indeed. And the problem with genomics techniques (and other omics techniques) is that, without a reasonable prior probability, the sheer volume of data magnifies the number of potentially spurious correlations by several orders of magnitude. Actually, even with a reasonable prior probability, the potential for spurious correlations is magnified by several orders of magnitude. That’s why correcting for multiple comparisons is so important that it’s built into most of the algorithms used to analyze this data.

  20. daedalus2u says:

    You can’t get a reasonable “prior” for this -omics discovery stuff because no one knows what is there, or what degree of complexity it takes to support a living organism and where that complexity is and how it gets transduced into the complexity that organisms exhibit. People’s intuition about stuff they don’t know or understand is extremely bad and usually wrong.

    As is usual, it isn’t what you don’t know that causes problems, it is what you know for sure that just happens to be wrong.

    You have to use a Bayesian approach, but you can’t apply it only a priori, you have to apply it at all times. The genome is a lot more fluid than (some) people suspected and a lot of the “highly heritable diseases” don’t seem to be genetic. They probably are not even diseases, but the default assumption that anything “bad” or “different” must be due to pathology is a very strong paradigm in many people, a paradigm that happens to be wrong in many cases.

    Dr Gorski, it isn’t “orders of magnitude”, it is orders of magnitude of orders of magnitude. In autism, the large GWAS have shown that for the familial type of autism there is no gene that accounts for more than a few percent of causation. If we assume that only 25 genes are important, 25 factorial is 10^25. If we assume that methylation at 25 sites is important too, 50 factorial is 10^64. Then there are environmental effects. It is easy to get hundreds and even thousands of orders of magnitude of complexity, or even more.

    The problem is that the development is a very robust process that usually goes in ways that are perceived to be “correct”. But we really don’t have any way of testing that hypothesis because development can’t be run multiple times. We know development is a process of multiple (many thousands) of coupled non-linear processes. They have to exhibit chaos and so must be differentially sensitive to changes in those parameters. They must be differentially sensitive to the butterfly effect, or using in vivo examples, differentially sensitive to the flagellum effect. Depending on how a particular flagellum moves, it will result in chaotic change in the phenotype of the organism that develops months later.

  21. DugganSC says:

    @micheleinmichigan:
    ^_^ I blame my family upbringing. My parents consistently corrected us immediately when we made a mistake and therefore my tendency is to do the same. But point taken that it generally does work better via emails.

  22. WilliamLawrenceUtridge says:

    In the past I’ve pointed out confusing errors in a comment and suggested my comment be deleted after it is corrected. Only for those that genuinely make the article less clear though, typos that are annoying but don’t actually impair your ability to understand aren’t worth following-up on.

  23. symball says:

    “Indeed. And the problem with genomics techniques (and other omics techniques) is that, without a reasonable prior probability, the sheer volume of data magnifies the number of potentially spurious correlations by several orders of magnitude.”

    One thing you didn’t highlight is that when used responsibly ‘omics’ studies are not regarded as definitive, they are just an early screen for potential causal, or diagnostic factors. First you trawl for interesting correlations, then you try too refine them and confirm the link, finally you prove them.

    Years of work and very often you will find that you have gone down a blind alley, and have to start again.

    not exactly what we see with these studies.

Comments are closed.