Oct 29 2012
NIH funds training in behavioral intervention to slow progression of cancer by improving the immune system
You are currently browsing comments. If you would like to return to the full story, you can read the full entry here: “NIH funds training in behavioral intervention to slow progression of cancer by improving the immune system”.
19 Responses to “NIH funds training in behavioral intervention to slow progression of cancer by improving the immune system”
I fully accept the criticisms laid out in this article, but I think one important line of criticism is missing from the critical commentary. The relevant section where I think it should have been placed is this:
“Andersen et al. do not report standard, unadjusted outcomes, such as a Kaplan-Meier estimate of the survival function. Their data reveal that the proportion of women experiencing a cancer recurrence did not differ significantly between the intervention condition (25.4%) and the control condition (29.2%; odds ratio, 0.83; confidence interval [CI], 0.46-1.48; P=.525). Moreover, there was no difference between the proportion of women who died in the intervention group (21.1%) versus the control group (26.5%; odds ratio, 0.74; CI, 0.40-1.36; P=.332); similar results are obtained if only those deaths caused by breast cancer are examined. Thus, the unadjusted analyses suggest that this is a null trial with respect to disease recurrence and survival.”
Just like it is important to separate “statistical significance” (i.e. low probability of obtaining data, or more extreme data, given that the null hypothesis is true) from “practical significance” (the difference is large enough to be on clinical importance), it is worth separating “statistically non-significant” (failing a significance test) from “clinically equivalent” (sample effect sizes and the range of plausible population effect sizes are the same). Confidence intervals should play a major role in any interpretation of research results, above and beyond just using it as another way to do a significance test.
If we look at the two confidence intervals here, we do not only note that they overlap zero (i.e. not statistically significant), but also that the upper arm of the confidence interval stretches quite high (to OR 1.48 and 1.36 respectively). This means that the range of plausible values for the population parameter includes values where the treatment actually does major harm to patients compared with control and a fairly large part of the error bar is located in the OR > 1 region.
So I would argue that saying that there is no difference between the two groups is not a strong enough criticisms. The treatment itself could very well be quite actively harmful compared with control (not just harmful as a side effect of leading to blaming patients if they relapse).
“My colleagues and I gave our now familiar argument that there was lack of evidence that any psychosocial intervention could reduce risk of recurrence and improve survival. There was also a lack of evidence for possible mechanisms by which such effects could conceivably be achieved.”
It seems to me that a plausible mechanism would be if a psychosocial intervention increased adherence to evidence-based therapies and this increased adherence reduced risk of recurrence and improved survival. Are you saying that is not a plausible mechanism, or that adherence is already so high that increased adherence would either have no effect or the effect size would be so small that it would take a huge trial to demonstrate it.
Thanks, marilynmann for your thoughtful comments.
I agree with you about the issue of clinical significance. However, in a highly speculative area of research that is starved for anything that can be claimed to be statistically significant, there is a tendency to settle for appearing to get that. I’ve attempted to compare differences in of immune parameters that investigators claim to be significant to what occurs in normal populations. Consistently, both intervention and control group in these PNI studies fall well within the normal range of values.
I think it’s quite easy to generate plausible mechanisms by which psychosocial interventions might influence adherence and to put them to unambiguous tests. There have been some efforts to educate postsurgical cancer patients and their caregivers concerning wound healing in the management of infections. There is some observational data that indicates that male cancer patients without partners interrupt the radiation treatment when they start to suffer side effects, something that is not observed with patients who have partners. This would seem to be a reasonable area in which to investigate the efficacy of psychosocial intervention, particularly once the reasons for the treatment interruption are established. But unfortunately PNI cancer researchers are focused and one may even say obsessed with trying to affect physical health outcomes by changing immune in hormonal parameters for which there is not well worked out mapping to these physical health outcomes.
Trick question: Someone tell me what they think the statistical test was for the part quoted by Emil Karlsson. Log-rank you say? Probably not, since a few paragraphs later “It is not possible to evaluate the claims of statistical significance made by Andersen et al. with respect to time to events without access to the data.” So we can’t even fit Kaplan-Meiers since we have no data? I admit that the failure to fit such models in the original paper is a show-stopper – I would have firmly declared it not possible to publish without that. I did not find their multivariate models or method of getting to them that strange, but in cases like this you really want to see the data yourself and make it pee in a cup, since you worry if the statistical methods are accurately described. I’m saying I am a bit concerned about the analysis, but could not find a giant offense. The design sucketh, that is certain – they are taking on way too diverse a set of patients and consequently have zillions of covariates to worry about. I also don’t like the control treatment – can’t ya take them fishing or give them some reading materials saying to watch diet, exercise and stay on your treatments. Without that the conclusion might be that doing anything to try and help may in fact help more than doing nothing, when I hoped the question is what that “anything” should be.
Someone tell me what “time by treatment interaction” means. What time? Time to event? If so, that’s usually on the left side of the model. Does it mean what year patient presented instead?
Being hopeless (and untrained at any rate) at statistics, I read this blog to get experts to interpret them as used in studies, for me.
The gist of this seems to be yet another effort to embellish cancer treatment with unproven rituals that, while they may make people “feel better”, have no scientific basis.
It is widespread in our culture to see cancer as an “enemy”, to be “battled” and “defeated”, in a “war”. Death amounts to “losing” the battle. I notice this even in commentary about Lance Armstrong’s fall from grace. Many commenters bemoan the sad situation of someone who “beat cancer” to end up being disgraced–as though it is cosmically unfair. These ideas are widespread and it is not that surprising, given the general encroachment of SCAM into medicine that these ideas would show up among doctors and other medical professionals as well.
Educating the media should remain a high priority for skeptics. Critics must remain vigilant and it is fortunate that you persevered and prevailed with yours. Getting this into the mainstream, though, seems even more difficult.
Thanks for bringing this to our attention. Brian Engler and I have just submitted a paper detailing the fall out onto the community level of the R25 Grants awarded by NIH’s Center for complementary and alternative medicine, NCCAM. These grants installed teaching of Alternative Medicine into respected Medical Schools.
It’s a shock to realize that this thinking has affected funding awarded by NCI. I checked there were two NCI awards for this project, 2011 and 2012. Both were for over $317 thousand dollars. Its surprising that our National Cancer Institute has diverted $700 thousand dollars from its research agenda to this unproven alternative medicine concept.
Eugenie Mielczarek
It is so gratifying to post my blog’s at Science Based Medicine and get responses, recall that I’m a refugee from the Psychology Today Blog, from which I bolted after they changed my title of a blog so that didn’t annoy a sorry, dvertises from Pharma. Sorry, Emil, I misattributed your smart comment to Marilynmann who had a smart comment of her own.
Eugenie, would love to see your paper when it is available.
I shoulda complimented Coyne and the co-authors for their efforts in publishing critiques. It takes guts.
Eugenie: I think we get to blame congress, for setting up NCCAM and it’s money. That the money so ear-marked then gets spent on CAM is rather expected, though one hopes they would put it in projects that actually do hold promise ( – as the munchkin says about their decedents “if any”).
James
Thanks for your interest in our study of NCCAM R25 Grants it’s grinding it’s way through a review process. Our previous study of NCCAM research grants was published in THE SKEPTICAL INQUIRER. Unfortunately print journalism is only interested in heart warming reports of mind- body interactions which persuade persons to forgo evidence-based medicine. The media is unable to recognize bad studies.
Eugenie Mielczarek
I don’t understand why it is OK for them to throw in adherence counseling (which I thought was generally already shown to work to some degree) only into the test group along with new or experimental mind-body stuff (which might not work at all). Isn’t that basically cheating? Differing compliance rates are normally something you would consider to be a confounding factor and want to AVOID, and yet in this experiment they are actively encouraging only the test group to comply and counting that as part of their experimental treatment.
If adherence counseling is already known to improve adherence to standard care by 5% (purely guessing), how is the intentional lack of adherence counseling in the control group any better scientifically than giving equal adherence counseling, but canceling the standard care prescriptions of a portion of the control group?
The paper says they measured drug levels to assess adherence rates (and then I suppose control for that mathematically), but why even include that at all? Adherence seems to have absolutely nothing to do with the other unspecified mechanisms they are hypothesizing about stress and immune system stuff. There seems to be no legitimate reason to group them together when you would logically have to try to split them apart again before drawing any conclusions.
@James Coyne:
Thanks for your articles on this.
“it was initially rejected, with the editor citing a standing policy of not accepting critical commentaries if authors refused to respond.”
Do you think it would be worthwhile for those of us without academic positions who spot problems in papers to try to push against these sorts of policies in the way that you did? Or is this something that is going to have to come from those within the academic system?
“I cannot imagine a NIH administrator making a similar argument for a large scale study of an herbal product or coffee enemas or other intervention with a similar undocumented mechanism of influence.”
Some treatments might be instinctively dismissed, but for psychosocial interventions, even when the proposed mechanism has been debunked, it seems that spun results from RCTs can still be used to pragmatically justify the promotion of treatments as ‘evidence based’.
@Janet: “The gist of this seems to be yet another effort to embellish cancer treatment with unproven rituals that, while they may make people “feel better”, have no scientific basis.”
I think that there’s also a real danger that RCTs can show certain treatments leading to improvements in certain questionnaire scores without necessarily making patients really feel any better. eg: People can just try to be positive and polite because they know that a therapist has tried to help them; or some psychosocial interventions seem to target the way in which patients use language, or think about problems, in a way that would be expected to alter their questionnaire answers regardless of whether it had any impact upon how they ‘really’ felt.
@geo
RE Do you think it would be worthwhile for those of us without academic positions who spot problems in papers to try to push against these sorts of policies in the way that you did? Or is this something that is going to have to come from those within the academic system?
I think you use up a certain amount of social capital in pushing back and risk alienating editors….unless you succeed. I have succeeded a number of times, but it is usually because I convinced editors honoring my request to change policies or to publish my commentary is a win-win outcome for me, but also for the journal or science. But some editors do not care about science or care about it less than professional politics.
@ConspicuousCarl
I agree that throwing adherence counseling into an already complex intervention makes it difficult to interpret effects, especially when the adherence counseling increases exposure to medical treatments. Elsewhere I talk about it as introducing co-treatment confounds. But an intervention like C2H is so confused in its goals and the investigators are so desperate to have something positive to claim, they will try lots of diverse strategies at once, even if they cannot explain results.
@Geo
I am learning how blogging, Twitter, and other alternative media can be effective tools for overcoming bad practices in the conventional journals, including their rejection of any post publication peer review.
@ James Coyne. Good luck with trying to spread the word and improve practices. It is easy for people to slip into a unjustified faith in things which purport to be evidence or science based, and this can have significant political and moral implications. Up until relatively recently I would tend to assume that claims being made in a paper were supported by the evidence cited, without taking the time to check – since I’ve started looking in more detail at the specifics of a lot of papers, I’ve come to see how often this is not the case.
[...] promotion. Whether cherry-picking the evidence, tooth fairy science, lack of plausibility, making inappropriate conclusions from studies, or other insufficiently rigorous methodology, bad science can be found at all levels of CAM. At [...]
Thank you for your well-referenced article on this topic. I think it’s very important and the issues you bring up reflect what seems to be a growing conflict under the surface between the political & career pressures and scientific honesty that permeates (bio)medical research in the US. The pressure to get grants funded which must be responsive to NIH’s priorities, to get grants funded that peer reviewers approve of, to publish as much as possible, to publish as quickly as possible, and the publication bias to positive results are factors that contribute to the persistence of these non-scientific interventions being promoted and researched. There are three areas that I’d like to comment on: 1) Perfectly legitimate areas of PNI research, 2) Growing pains and historical/political/career problems in neurovirology, and 3) Affirm the areas of great concern that you have outlined.
I attended the Psychoneuroimmunology Research Society (PNIRS) annual meeting this year (which was local to me), because I was invited to give a lecture as part of a short educational course on molecular biology. (The lecture was on the basics of microRNA biology, by the way). Much of the research that I saw being presented was derivative from Sapolsky’s work on stress, cortisol, glucocorticoids, and the immune system; along with the “glucocorticoid cascade hypothesis” of depression/mood disorder. Within this hypothesis are perfectly plausible biological mechanisms related to hypothalamic/pituitary/adrenal (HPA) axis dysfunction having physiological consequences on various organ systems, including the immune system. Cortisol, for example, is an immunosuppressive molecule. Sleep disturbance and metabolic malfunctions (as a result of stress hormone signaling) are plausible mechanisms for immune dysfunction.
(For what it’s worth, my dissertation research was on the kinetics of glucocorticoid receptor signaling and chaperone proteins in neurons; which are affected by pro-inflammatory paracrine signaling).
At this PNIRS meeting, researchers from my home institution presented posters related to HIV-infection, cognitive measurements, and correlations with pro-inflammatory cytokines measured in the cerebrospinal fluid. There is a long history of studying effects of HIV on the brain since the beginning of the epidemic because there was, in fact, HIV “encephalitis” characterized in the era before highly active antiretroviral therapy (HAART) was the norm in the US. There were also a range of opportunistic infections that affected the brain. In the early days, a prolific research community was built to characterize the neuropathology of HIV infection and understand the neurocognitive or “behavioral” outcomes in these patients. This was very useful in understanding the biology of HIV and the clinical course of end-organ disease and AIDS. A significant proportion of this research was funded by the NIMH. The result is a large contingent of academic scientists who are now at the stage of full-professor or tenure and research infrastructure around this field; in departments that mainly focus on brain & behavior; who depend on funding from NIMH, and who are attempting to continue to study HIV & and the central nervous system in this capacity. The problem (as I see it) is now that HAART is the norm for HIV patients in the US, and we can suppress viral loads to undetectable levels within months of initiating therapy, and a rebound of CD4 cells — these neuropathology and behavioral outcomes for HIV patients are becoming irrelevant. But the focus of this “old guard” is still there, and they are continuing to train and nurture young scientists in this field. Because HAART is rendering these neuropathology outcomes irrelevant, they are switching to these other NCCAM-type studies referenced in your post, the so-called psychosocial interventions and HIV clinical course outcomes. (Thankfully, none of them came from my home institute).
There is a resurgence of interest in HIV in the central nervous system currently — but in the context of identifying potential cellular reservoirs of latent provirus — with the intention of eventually developing strategies for eradication & cure. However, because of the source of funding (NIMH), and the past research focus of the existing power brokers in the field (and NIH program officers are complicit participants to this), the clinical outcomes & interventions are all these “behavioral” domains that are rarely clinically relevant, and we have to stretch very hard to even measure. I do not see an easy way out of this entrenched mess. When I propose certain ideas to one group (say, NIMH or NIDA) — it’s not interesting because I need to have a mental health outcome; if it’s proposed to another group (say, NIAID) — it involves the brain, and therefore should go to the NIMH.
All that said — I’d like to shift gears to a final comment on some of the things I saw at the PNIRS meeting. Definitely, I saw a bias toward reporting non-significant results as positive findings. I do not understand the relationship between PNI & cancer. Certainly there is a link between central nervous system, stress, and immunology — but the way that some interventions are being proposed (mental health interventions for an immunology disease) seems to ignore plausible biological mechanisms. It was also distressing to see some interventions (say Tai Chi or acupuncture) reported as having positive results on nearly anything, while ignoring placebo effect (or worse, designing the control group to not get a placebo). Some (not all, I should qualify) in the room seemed to really truly believe that fixing meridians and vital life forces and energies and such really do positively impact the clinical course of almost any disease. I think they are stretching to the immune system as a biological mediator solely for practical purposes. The assays are easy and inexpensive to do, and blood serum and cells are easily obtainable specimens from patients enrolled in a study. That’s a pragmatic view. I think that PNI research is being infiltrated by CAM simply because CAM researchers need to find or study a “mechanism” and immunology measurements are generally safe, cheap, and easy to do.
I may be overly generous to the founders of the PNI field (I am relatively young and a new investigator); but having attended their meeting, I observed that there seems to be various camps who talk past each other. In the future, I hope to have the bravery to stand up to bad science (at the risk of my career or future publication prospects).
Excellent, well-informed post, etatro, that serves as a nice supplement and qualification of mine. Please keep the reports coming from the PNI camp.
I almost missed this article. I’m glad I didn’t. It’s a bit over my head, but I think it was worth the effort or getting what I could from it and the comments helped.
I have a laymen’s question that is tangential to this article. I often hear about the effects of emotional “stress” on the immune system, in terms of inflammation, illness, autoimmune disease, heart disease, etc, but I don’t have a good sense of how evidence based any of these claims are. It does seem well established that emotional stress can trigger migraines and flares in people with auto-immune disease, some skin reactions, but beyond that what’s up?
Conversely, is there evidence that any current stress management techniques lower occurrences of symptoms in illnesses know to be triggered by stress?
Sorry, I know this is broad, but I just thought I’d throw it out there, maybe for a future post or two.
Very good question, #mousethatroared, I think the role of stress and the immune system are much better charted in cardiovascular disease than in cancer, for instance. For instance, a generalized immune response has negative implications for artery disease. Fori instance, maybe untreated gum disease can have implications for cardiac problems.
The problem with claims about psychological interventions affecting the immune system in ways beneficial to physical health is that the changes in immune functioning that occur are modest, if they occur at all.
etatro makes some excellent points above, without being as skeptical as I am.
Efforts to affect cardiovascular outcomes by treating depression have been disappointing. Depression in cardiac patients can be treated with the same effectiveness as depression in people without cardiac problems, but there is no apparent effect on the likelihood of another heart attack or death from cardiac disease.