Articles

The perils and pitfalls of doing a “vaccinated versus unvaccinated” study

The anti-vaccine movement is nothing if not plastic. It “evolves” very rapidly in response to selective pressures applied to it in the form of science refuting its key beliefs. For instance, when multiple studies looking at the MMR vaccine and autism failed to confirm the myth that the MMR causes autism or “autistic enterocolitis,” most recently late last year, it was not a problem to the anti-vaccine movement. Neither was it a major problem to the movement when multiple studies similarly failed to find a link between mercury in the preservative thimerosal that used to be in most childhood vaccines and is no more (except the flu vaccine) and autism. No problem! Andrew Wakefield is alleged, based on strong evidence, to have falsified his data alleging a link between the MMR vaccine and “autistic enterocolitis”? Fuggedabouddit! The anti-vaccine movement simply pivoted neatly, de-emphasized points that the evidence was so clearly against that even they couldn’t spin it to a positive anymore, and found new bogeymen. These days, it’s the “toxins” (such as formaldehyde and the latest antivax bogeyman, squalene), and “too many too soon” (a gambit given seeming respectability by Dr. Bob Sears and Dr. Jay Gordon, apologists for and supplicants to the anti-vaccine movement both.

However, there is one trait of the anti-vaccine movement that, however its camouflaging plumage may evolve, never, ever changes. It is as immutable as believers say that God is. That trait is that, whatever other claims, the anti-vaccine movement makes, at its core it is always about the vaccines. Always. No matter how often science fails to find a link between vaccines and autism or vaccines and whatever other horreur du jour the anti-vaccine movement tries to pin on vaccines, no matter how many studies do not support the viewpoint that vaccines cause autism, no matter how much the anti-vaccine movement tries to deny and obfuscate by saying that it is not “anti-vaccine” but rather “pro-safe vaccine,” at its core the anti-vaccine movement is about fear and loathing of vaccines. Always. When inconvenient science doesn’t support their views, anti-vaccine activists either ignore the science, distort the science, or launch ad hominems against the people doing the science or citing the science. And, as I said before, the claims of the anti-vaccine movement evolve. Never again will the anti-vaccine movement make the horrific mistake of yoking itself to a hypothesis that is as easily testable as the hypothesis that mercury in vaccines causes autism. The claim that mercury in vaccines causes autism predicted that, if thimerosal were removed from vaccines or reduced to pre-“epidemic levels” of the early 1990s, then autism rates should plummet. Thimerosal was removed from nearly all childhood vaccines (the sole exception being some flu vaccines), reducing infant mercury exposure from vaccines to levels not seen since the 1980s; yet autism rates continue to rise. This is about as resounding a refutation of the hypothesis that mercury in vaccines is a major cause or contributor to autism that even the anti-vaccine movement has backed away from the pure claim, which has now evolved to unnamed “environmental toxins,” either in concert with mercury or with other nasty things, as being the Real One True Cause of Autism.

It’s evolution in action. These new claims are much “fitter” because they are much harder to falsify through scientific research, epidemiology, and clinical trials.

To understand the persistence of the myth that vaccines cause autism, we must consider the myth underlying it, namely the myth of the “autism epidemic.” It is certainly true that the number of autism diagnoses has been increasing at a seemingly inexorable rate ever since the early 1990s. At around the same time, multiple new vaccines were introduced to the schedule. Also around the same time (namely 1994), the diagnostic criteria for autism and autism spectrum disorders were greatly broadened. Also, throughout the 1990s and the 2000s, more resources were invested in screening children for autism. As is the case with other conditions or diseases, the more you screen for something the more you will find, often the less severe cases that would have been missed otherwise. In any case, a combination of diagnostic substitution, increased awareness, increased screening, and increased support services that brought revenue to school districts to help autistic children account for the vast majority, if not all, of the increase in autism prevalence over the last 20 years or so. As Joseph puts it:

Nevertheless, it appears that when ASD is screened thoroughly in a population, or when there’s a lot of awareness and good ascertainment, prevalence is found to be closer to 1%. This is not new. The following is what Lorna Wing and David Potter said on the subject as early as 1999.

Indeed, recent NCSH data do not show any evidence of an “autism epidemic.”

However, the key to the myth that vaccines cause autism is the observation that autism prevalence has increased rapidly since the early 1990s, and that was around the same time when the vaccination schedule was expanded to include more vaccines. Of course, as I’ve pointed out before, this is a classic case of confusing correlation with causation, and, as I’ve also pointed out before, one could point to a number of other things that happened in the 1990s that correlate with increased autism diagnoses. For example, the first easy-to-use web browsers were first introduced in the early 1990s, leading to an astonishingly rapid growth of Internet use outside of universities and work environments. Using the same sort of logic as anti-vaccinationists use, we could just as easily blame the Internet for the “autism epidemic”! In any case, thanks to the human tendency to confuse correlation with causation and desire to have an explanation for anything we consider scary, the myth that vaccines cause autism persists among even highly intelligent people who should know better. Indeed, one person at a recent skeptical meeting I attended was clearly very suspicious of vaccines and not so clearly reassured by what fellow SBM blogger Peter Lipson and I explained. Consistent with the persistence of this myth, refutations of the contention that vaccines cause autism are usually either met with denial or the classic false dichotomy, “Well, if vaccines don’t cause autism, what is causing this huge increase in autism?” as if vaccines were the only plausible option.

Thus, I think, was born the “too many too soon” gambit, in which the anti-vaccine movement, so wedded to the idea that vaccines must cause autism, have latched onto the idea that somehow it’s the “overvaccination” or “too many vaccines.” According to this idea, somehow excessive “immune stimulation” or injection of “toxins” that is leading to and “epidemic” of autism, asthma, and a huge variety of other ailments and conditions.

This “too many too soon” chant has lead to a demand by the anti-vaccine movement that the government conduct a large study of “unvaccinated” versus the “vaccinated” children to compare them for health outcomes and, especially, the prevalence of autism. The expectation, of course, on the part of the anti-vaccine movement is that such a study would show that unvaccinated children are far healthier and don’t suffer from nearly the same rate of autism. I don’t think that people like J.B. Handley realize how risky their gambit is. Such a study would have a very high risk of torpedoing virtually everything the anti-vaccine movement has been working toward in terms of promoting their message of fear about vaccines as being somehow credible (or at least not unreasonable) and based on science (more on that later).

None of these concerns has stopped the drones over at the happy home for anti-vaccine propagandists (Age of Autism) from busily promoting the idea of a “vaxed versus unvaxed” study and, as is their M.O. for any critics, harshly disparaging anyone who dares to point out the difficulties and lack of scientific rationale for such a study, the latest target being Tom Insel of the Interagency Autism Coordinating Committee (IACC), who recently testified in front of Senator Tom Harkin’s comittee (yes, that Tom Harkin).

The misunderstanding and misrepresentation of epidemiology being trotted out to justify this study are astounding. For example, last week on AoA, a mechanical engineer named Catherine Tamaro tried to lecture scientists on statistics and the difference between observational versus experimental studies using quotes from a statistics textbook. She began:

After reading Katie Wright’s summary of the August 3 Senate Committee on Appropriations’ hearing on autism (HERE), I would like to comment on Dr. Insel’s testimony to Senator Harkin on why a study comparing vaccinated vs. unvaccinated children has not been done to date. Dr. Insel claimed that such a study would be “unethical” because, as he apparently envisions it, the study would entail dividing a large cohort of newborns into two groups, vaccinating one cohort but not the other through age two, and then comparing outcomes. This is just a bit disingenuous and I would like to explain why.

Of course, Ms. Tamaro is either ignorant or disingenuous herself in that some anti-vaccine advocates do indeed call for just such a study, even going so far as to demand a randomized, double-blinded study. J.B. Handley himself has attacked people who correctly call demands for such a study “unethical.” Apparently Ms. Tamaro isn’t aware of the strain running through the anti-vaccine movement that does want just such a study, or she herself is being even more disingenuous than she accuses critics of being. Tamaro then continues:

Research studies are divided into two categories, observational studies and experimental studies. An observational study observes individuals and measures variables of interest but does not attempt to influence the responses. (The “epidemiological” studies to which Dr. Insel refers are actually observational studies.) An experimental study, on the other hand, deliberately imposes some treatment on individuals in order to observe their responses; the purpose of an experiment is to study whether the treatment causes a change in the response.

This paragraph just goes to show how a little knowledge is a dangerous thing. There are many flavors of “observational” studies, with varying degrees of power to detect differences and varying degrees of ease with which confounders are accounted for. These include cohort studies (both retrospective and prospective), cross sectional studies, longitudinal studies, and case series. All have their strengths and weaknesses, and the studies of vaccines and autism have used several kinds of methodology. Not that that sways Tamaro:

All studies done to date investigating a correlation between vaccinations and autism have been observational studies, but no observational study has been done comparing the prevalence of autism diagnoses in a vaccinated human population compared to an unvaccinated human population. When Dan Olmsted points out that he has identified large populations of unvaccinated children in the United States and asks why a study has not been done on them, he is actually asking why an observational study has not been done. When Senator Harkin asks Dr. Insel why a study has not been done on vaccinated vs. unvaccinated American children, he too is actually asking why an observational study has not been done to date. Dr. Insel, however, chooses to respond by saying that an experimental study would be required in order to resolve the issue.

Tamaro may have a bit of a point about Insel choosing the easiest to attack, but she is doing the same thing by ignoring the fact that there have been calls from the anti-vaccine movement for experimental studies, which, of course, would be highly unethical because they would leave large numbers of children completely unvaccinated and thus vulnerable to vaccine-preventable diseases. In any case, here’s where Tamara goes right off the deep end:

I would like to point out the epidemiological similarity between smoking/lung cancer and vaccines/autism. Smoking has been proven to cause lung cancer, yet not a single experimental study on humans was ever done – all of the human studies proving that smoking causes lung cancer were observational. The experimental studies were performed on research animals only. Attached at the end of this letter is a lesson taken verbatim from an introductory course in college statistics describing how the connection between smoking and lung cancer was made.

There have been lots of experimental studies on research animals of vaccines trying to show a link between vaccines and autism. I’ve written about some of them right here, and other bloggers have discussed them in detail as well. For example, there was the infamous Mady Hornig “rain mouse” study, in which she claimed that thimerosal at the doses infants received, adjusted for size, caused autistic symptoms. Both Prometheus and Autism Diva enumerated the numerous flaws and ethical lapses in that experiment. Then there was the more recent (and even more unethical) Laura Hewitson experiment looking at vaccinated and unvaccinated Macaque monkey infants. I was appalled at how badly designed and grossly unethical that experiment was, not to mention at the enormous undisclosed conflicts of interest of the investigators. The problem, of course, is that there is not yet a good animal model of autism. If there were, you can bet that there would be a huge amount of research using animal models. It’s not because the NIH is unshakably opposed to animal research on vaccines and autism. Rather it’s because such research is inherently not particularly useful or feasible absent a good animal model or two. Moreover, the history of such research (i.e., Hornig and Hewitson) is not exactly cause for optimism, given how badly done these studies were.

In addition, the tobacco comparison is a favorite canard of anti-vaccinationists (for instance, Dr. Jay Gordon). Certainly, it was primarily epidemiological studies that showed that smoking causes cancer. While she is correct to say that an experimental (i.e., randomized, blinded) study is not always necessary to provide sufficient evidence of causation to conclude that there is causation, she’s picked the wrong example for a number of reasons. For smoking and cancer, the association is very strong, stronger than almost any environmental toxin and cancer that I can think of off the top of my head. Smoking results in a ten-fold increased risk of lung cancer. Such a high magnitude of increased risk is generally not that hard to detect with an epidemiological study of even fairly weak design, as long as there are sufficient numbers and the condition is not extremely uncommon. But more importantly, what Ms. Tamara either doesn’t know or fails to acknowledge is that the very same sorts of studies that identified smoking as a huge risk factor for lung cancer and cardiovascular disease are the same sorts of studies that have failed to find an association between either thimerosal and autism or vaccines and autism. For example, it was a case control study by Sir Richard Doll that was credited with first detecting the link between smoking and lung cancer (although actually Nazy doctors discovered the link with a case control study two decades before Sir Doll did).

In any case, Ms. Tamara is also wrong when she says that a study of the vaccinated and unvaccinated has never been undertaken. First, there have been studies looking at who the unvaccinated are and where they live. One study published in Pediatrics in 2004 found that:

Undervaccinated children tended to be black, to have a younger mother who was not married and did not have a college degree, to live in a household near the poverty level, and to live in a central city. Unvaccinated children tended to be white, to have a mother who was married and had a college degree, to live in a household with an annual income exceeding $75 000, and to have parents who expressed concerns regarding the safety of vaccines and indicated that medical doctors have little influence over vaccination decisions for their children. Unvaccinated children were more likely to be male than female. Annually, 17 000 children were unvaccinated.

In other words, there are a lot of difficult confounders to control for, and the number of unvacccinated children is small, further complicating epidemiological studies. Ironically enough, Generation Rescue itself tried to do it. It was a very bad study, really no more than a phone survey, and its results were amusing in that they found that the highest rates of autism were in the so-called “partially vaccinated” children. The clear implication of the study was drolly described by Kevin Leitch:

There’s no getting away from this. This is a disaster for Generation Rescue and the whole ‘vaccines cause autism’ debacle. Generation Rescue’s data indicates that you are ‘safer’ from autism if you fully vaccinate than partially vaccinate. It also indicates that across the spectrum of autism, you are only 1% more likely to be autistic if you have had any sort of vaccination as oppose to no vaccinations at all – and thats only if you are male. If you are a girl you chances of being on the spectrum are less if you have been vaccinated! Across both boys and girls, your chances of being on the spectrum are less if you have received all vaccinations.

I can only conclude that Ms. Tamara is also quite naive in that she clearly has no clue just how much money and how many children an observational study of the vaccinated versus unvaccinated would require to do properly, much less how tricky it would be to control for confounders, given that the unvaccinated vary in significant ways from the vaccinated. Skeptical blogger extraordinaire Prometheus tells the tale. First, he points out how few completely unvaccinated children there are to study, perhaps around 50,000 in the entire U.S., in the 3-6 year old age cohort that would be most fruitful to do a study looking at autism incidence in the vaccinated and unvaccinated. Prometheus then explains:

Let’s say – for the sake of argument – that we decide that a 10% difference in autism prevalence is enough to convince the skeptics that vaccines might cause autism and that a less than 10% difference will convince the believers that vaccines don’t cause autism. [I know, the latter assumption is pure fantasy.]

Well, plugging those numbers in – along with the current 1 in 150 autism prevalence – we find that we need over 360,000 children in each group to detect a 10% difference (you can try it yourself here). Unfortunately, that is more than the total number of unvaccinated children in the US, so that’s not going to happen.

What can we get with our “sample” of 49,652 unvaccinated children? If we manage to include each and every unvaccinated child in the US in the study, we could detect a 26% or more difference in autism prevalence.

Of course, it’s not even remotely practical to expect to get 100% of the unvaccinated children in the country into a study. How more about a practical number – say, 10% of them? That would allow us to detect a 70% or greater difference – about a three-fold difference in autism prevalence between the fully vaccinated and unvaccinated groups.

Does anyone here think that parents who fervently believe that vaccines cause autism would accept negative results from a study that’s only powered to detect a three-fold difference in autism rates between the vaccinated and unvaccinated as sufficiently reassuring to accept the current vaccination as safe? Prometheus was right when he said that a study powered to find a 10% difference would probably not sufficiently reassure them. Given the religious fervor with which the anti-vaccine movement clings to the myth that vaccines cause autism, I doubt that it would accept a negative result from a study powered to detect a 1% difference in autism rates as sufficiently reassuring to abandon its fear. Moreover, as Prometheus tells us, even the study described above would be inordinately expensive and difficult to do. So he did a “back of the napkin” calculation of what a more feasible study could accomplish:

Finally, let’s “run the numbers” on a more practical study – one where we are able to enroll 500 unvaccinated children and 5000 fully vaccinated controls matched for age, sex, socioeconomic group, geographic location, urban vs rural vs suburban setting and race.

This study – which would still be very difficult and expensive to do – would only be able to detect a more than 15-fold difference in autism prevalence between the two groups. It could detect as little as a 7-fold difference, but only if we were willing to accept a beta error (chance of erroneously saying there is no difference when there is a difference) of over 50%.

I doubt this would “satisfy” the vaccines-cause-autism believers if the results were negative.

I can’t help but note that the study described by Prometheus would probably fail to find the well-known increased risk of lung cancer and heart disease due to smoking, the more so since the incidence of lung cancer in nonsmokers is considerably lower than 1 in 150, which is how many children are estimated to be autistic. Remember, the relative risk of lung cancer due to smoking is on the order of ten-fold.

The only way to get around the problems inherent in designing a study with sufficient numbers of unvaccinated children of sufficient power to detect a difference in autism prevalence between the unvaccinated and vaccinated children small enough to reassure most parents that vaccines do not cause autism would be to expand the study to multiple nations. Of course, doing such a study would be even more enormously expensive, take several years, and, because funding for autism research is pretty much a zero sum game, would divert huge amounts of money from more promising research to chasing down a highly implausible hypothesis that has virtually no credible empirical support behind it, either from basic science, epidemiology, or other evidence, certainly nowhere near enough evidence to justify such a huge expenditure and effort. Alternatively, as Prometheus points out, we could look at dose-response models:

Of course, we wouldn’t have to just look at unvaccinated vs fully vaccinated with this study, which is a large part of its superiority. We could look at a dose response of vaccination – to see if it really is “too many” – as well as the age at youngest vaccination – to see if it really is “too soon”. In fact, a few studies have already looked at those issues and found that there is no difference between the autistic and non-autistic groups. I suspect this is the reason the folks pushing to “put on a study” want to look at vaccinated vs unvaccinated – they hope that the numbers will be different (or, at least, not as definitive) the other way round.

In fact, I rather suspect that the smarter among the anti-vaccinationists know all the problems inherent in doing a study of vaccinated versus unvaccinated children. Certainly the government does, hence its reluctance to spend all sorts of money chasing a highly improbable hypothesis. (If only it would apply that reasoning to NCCAM!) In reality, the “vaccinated versus unvaccinated” gambit is just that–a gambit. The leaders of the anti-vaccine movement probably know that doing a study with sufficient power and numbers to exclude even a modest risk of autism due to the current vaccine schedule is so expensive and impractical that it would probably never be done and that smaller studies that are feasible will have too little power to reassure those who believe that vaccines cause autism that vaccines are in fact safe. Why do it then? It keeps the troops fired up thinking that there is some huge conspiracy to prevent such a study because of the fear of its results or that the government just doesn’t care enough about autistic children to do such a study.

On the other hand, antivaccinationists should be very careful what they ask for. They may just make enough of a pain of themselves to get it. True, getting the resources necessary to do a study the like of what Prometheus described intially would monopolize autism research funding for years, but the anti-vaccine movement doesn’t really care about that, because it’s always been all about the vaccines more than helping autistic children. Worse, if the government ever did spend the money on such an enormous study and it was resoundingly negative, it’s easy to predict that it would make no difference. As they have done before for other large studies, anti-vaccinationists would discount the results and cry bias. Still, if the government caves and decides to do such a study, it is up to us in the scientific community to make sure that it’s done by no one but the best epidemiologists, in other words, that it’s a proper study that correctly controls for confounders and can answer the question being asked, not the dubious study custom designed to have the maximal chance of a false positive result, which is of course what the anti-vaccine movement really wants.

Posted in: Clinical Trials, Medical Ethics, Public Health, Science and Medicine, Vaccines

Leave a Comment (56) ↓