Last week, I wrote one of my characteristically logorrheic meandering posts about what turns a scientist into a crank or a doctor into a quack. In a sort of continuation of this line of thinking, this week I’ll turn my attention to one of the other most common characteristics of a crank, be he scientific crank (i.e., a creationist), a quack, or historical crank (i.e., Holocaust deniers), specifically how he views the peer review system.
Not suprisingly, one of the favorite targets of pseudoscientists is, in fact, the peer review system. Indeed, it’s a very safe thing to say that, almost without exception, cranks really, really, really don’t like the peer review system for scientific journals and grant review. After all, it’s the system through which scientists submit their manuscripts describing their scientific findings or their grant proposals to their peers, and their peers make a judgment whether manuscripts are scientifically meritorious enough to be published and grant applications scientifically compelling enough to be funded. Creationists hate peer review. HIV/AIDS denialists hate it. Anti-vaccine cranks like those at Age of Autism hate it. Indeed, as a friend of mine, Mark Hoofnagle pointed out a couple of years ago, pseudoscientists and cranks of all stripes hate it. There’s a reason for that, of course, namely that vigorous peer review is a major part of science that keeps pseudoscientists from attaining the respectability that science possesses and that they crave so.
Far be it from me to get all Panglossian on you and claim that the peer review system is the “best of all possible worlds,” or anything like that. Having participated in the system at multiple levels, including at the receiving end, as a peer reviewer for journals, and as a reviewer on a study section, I know there’s no doubt that the system has problems and could do with considerable improvements. However, when I hear rabid bashers of peer review characterize it as “crony review,” I tend to echo Winston Churchill’s famous statement about democracy that peer review is “the worst way of determining what science should be published and funded except for all those others that have been tried.” Certainly, the bashers can’t seem to propose something better. Of course, that’s because the real purpose behind the numerous criticisms made by cranks about peer review is not to reform or improve the system in order to prevent bad science from being published and funded and good science from being rejected, but rather to weaken the system or alter it so that they can get their favorite pseudoscience published an/or funded, thus allowing them to attain the respectability of real science that they so crave. In other words, it is important in assessing attacks on the peer-review system to determine whether the criticism is designed to suggest real changes that might improve it or, as is the case with most cranks, to tear it down in order to make it more amenable to accepting their pseudoscience.
If there is one sterling example of this phenomenon that I’m aware of, it’s an article that I first encountered a couple of years ago by a surgeon at the University of Washington named Donald W. Miller, Jr. (website). I remember coming across it while delving into the blogs and websites of HIV/AIDS denialists. Not surprisingly, Dr. Miller is a regular columnist at LewRockwell.com, where he has referred to HIV/AIDS denialist hero Peter Duesberg as a “modern day Copernicus” and questioned the HIV/AIDS hypothesis as an “orthodoxy.” Not surprisingly, Dr. Miller is also an anti-vaccinationist, shamelessly promoting the myth that thimerosal in vaccines was a cause of the “autism epidemic” (it wasn’t; it’s a failed hypothesis) and that anthropogenic global warming is a “scam.” I first became aware of this article, The Government Grant System – Inhibitor of Truth and Innovation? on the HIV/AIDS denialist blog You Bet Your Life. It also appeared on LewRockwell.com under another title, The Trouble With Government Grants. In the article, Dr. Miller launched into a blistering attack on the current peer review system used by the NIH, an attack that revealed his profound ignorance of how government peer review systems (not system) work.
I knew there were going to be problems right away. First, Miller used the term “truth” in the title of one of the versions of his article. Science is not about “truth”; it is about understanding how the world around us works to as good an approximation as we can get. Worse, very early on in the article, Dr. Miller shows that he can’t seem to get his facts straight, mangling the concept of “triage” and making me seriously wonder if he has ever served on an NIH study section. Certainly, searching the CRISP database (the best way to find out if anyone has an NIH research grant, by the way), I could find no evidence that he has ever been the principal investigator or co-investigator on an NIH grant, which if true would more or less disqualify him from sitting on an NIH study section. I also can’t help but note briefly that Dr. Miller, like so many physicians and scientists who turn to the dark side of pseudoscience, seems to have had a respectable publication record in peer-reviewed journals as an academic cardiac surgeon until 1991, after which he has not published in peer-reviewed journals. (Note: Despite its claims otherwise, the Journal of American Physicians and Surgeons, as Dr. Miller has, does not count as a peer-reviewed journal, for reasons I have discussed extensively before.) In any case, here Dr. Miller describes the peer review system as he thinks it stands:
The Center for Scientific Review “triages” applications it receives. A cursory appraisal eliminates one-third of the applications from any further consideration, and it selects the remaining two-thirds for competitive peer review. CSR sends each application to a Study Section it deems best suited to evaluate it. Peers in Molecular Oncogenesis, Cognitive Neuroscience, Cell Structure and Function, Hematopoiesis, HIV/AIDS Vaccine, and 167 other Study Sections review grant applications. Each Study Section has 12-24 members who are recognized experts in that particular field. Members meet three times a year to review 25-100 grants at each meeting. Two members read an application and then discuss it with the other section members who collectively give it a priority score and percentile ranking (relative to the priority scores they assign to other applications). An advisory council then makes funding decisions on the basis of the Study Section’s findings, “taking into consideration the [specific NIH] institute or center’s scientific goals and public health needs” (Scarpa, 2006).
Not quite. This is what the NIH CSR says about peer review:
One or more CSR Referral Officers examine your application and determine the most appropriate Integrated Review Group (IRG) to assess its scientific and technical merit. Your application is then assigned to one of the IRG’s study sections. A study section typically includes 20 or more scientists from the community of productive researchers. Your application also will be assigned to the NIH Institute or Center (IC) best suited to fund your application should it have sufficient merit. (More than one IC may be assigned if appropriate.)
In reality, in the first pass through the CSR, Referral Officers do little more than (1) make sure the grant is formatted correctly (yes, they do check to see if you used a 10 point font instead of 11, shrank the margins beyond what the rules state, went over the page limit, or tried to get by without all the necessary institutional signatures, and if they find that you did any of those things or others the grant will not be forwarded to a study section); (2) verify that it fits the criteria for the grant mechanism being applied for; (3) check against an NIH database of grant applications to make sure that the applicant hasn’t submitted the same grant to two different funding mechanisms during the same grant cycle; and (4) figure out the most appropriate Integrated Review Group to send it to. I suppose it’s possible that 30% of scientists are too stupid or careless to follow the formatting requirements properly and to include all the needed information, but I doubt it. Even if that were the case, the scientists would have no one to blame but themselves; the instructions, although voluminous, are quite clear at least about the basic formatting requirements and page limits for an NIH grant application. In any case, pretty much every grant that’s formatted correctly and contains all the required elements is assigned to a study section for review.
During a study section, the appraisal of which grants are “triaged” is not “cursory.” Every grant application is assigned to approximately three reviewers (the number may vary, depending on the grant mechanism and study section). In the study section on which I last served, for example, every application was assigned to a two primary reviewers (Reviewer 1 and Reviewer 2) and a secondary reviewer (the Discussant). Both primary reviewers were expected to read the grant application in detail, write up a 2-4 page review of it, and assign it a proposed priority score. The discussant was also expected to read each grant assigned to him in detail but only to write up a briefer 1-2 page review and assign a proposed score. Everyone else on the study section tends to look over grants to which they are not assigned as Reviewer 1 or 2 or the Discussant, other than perhaps the abstracts of the grants, but that’s understandable, given that most reviewers are assigned around 10 grants to read in six weeks and that it can take several hours per grant to review it. (At least, that’s about how long it takes me.) At the study section meeting itself, to convene the meeting the chair listed the grants whose initial proposed priority scores were in the bottom half. Because these grants clearly had no chance at being funded during the cycle being discussed, they were then “streamlined” (or, colloquially, “triaged”), meaning that they would not be discussed in detail at the full study section. Potential streamlining candidates whose reviewers assigned them widely divergent scores, indicating a sharp disagreement over their scientific merit, were often specifically pulled aside for discussion before voting on streamlining. Indeed, if either reviewer was insistent about it such grants would usually be discussed before the whole study section despite their low average priority score, and if any study section member strongly objects to the streamlining of any grant application, it was discussed.
Unfortunately, given the largely increased number of grant applications and the down economy, which has led to NIH budgets that, until recently, have not even kept up with inflation, the percentage of NIH grants funded steadily fell from 2004 to 2008. At the low point, for some Institutes, less then 10% of grants were being funded. Consequently, often more than 50% of grants were triaged during that period, because any grant that didn’t reach at least the 30th percentile had no chance. It wasn’t even “in the zone,” so to speak. Fortunately, that trend is starting to reverse, thanks to additional funding to the NIH, although where the new equilibrium will fall is anyone’s guess.
After streamlining, once the discussion of the remaining grants started, Reviewer 1 would usually lead it it for his assigned grants, with Reviewer 2 and the Discussant chiming in. After a full discussion, each member of the study section would then assign a score. Contrary to Dr. Miller’s distorted description, the only differences in treatment between “triaged” grant applications and those discussed at the full study section is that triaged grants are not discussed in detail (although they are reviewed in detail), and a “summary statement,” which boils down the written reviews and group discussion into a summary, which is (usually) highly useful for applicants in guiding revisions of the application for resubmission. Also included are the individual reviewer comments appended to the summary statement. For triaged grant applications, on the other hand, the three written reviews are returned to the applicant, who is free to revise and resubmit based on the comments of the three reviewers. It’s not quite as useful, but still helpful. Indeed, my first application for an NIH R01 grant was triaged, and the comments were helpful in my revising and ultimately succeeding the second time around. Moreover, study sections do not assign percentile scores, only priority scores. Percentile scores are generated from the Gaussian curve of all the priority scores, and it is the percentile score that determines which grants are funded.
The bottom line is that, whatever major faults the NIH grant approval process has, doing only a “cursory” evaluation of the applications at any stage is not among them.
It’s at this point that you find out where Dr. Miller was really coming from, and it wasn’t from the perspective of someone who wants to reform the system. Indeed, the system could use some reform. Whether the “reform” of a new scoring system implemented this year or proposals to cut the page limits for grant applications from 25 to 15 in order to force applicants to deemphasize methodological detail and emphasize broad research rationale and strategy will improve peer review remains to be seen. However, that’s not where Dr. Miller is coming from in this article. Rather, he comes from the perspective of a friend of pseudoscience who feels that the system doesn’t give his HIV denialist buddies a fair shake, which is evident in how he starts out semi-reasonable and then goes right off the deep end. Here’s the semireasonable part:
The grant system fosters an Apollonian approach to research. The investigator does not question the foundation concepts of biomedical and physical scientific knowledge. He sticks to the widely held belief that the trunks and limbs of the trees of knowledge, in, for example, cell physiology and on AIDS, are solid. The Apollonian researcher focuses on the peripheral branches and twigs and develops established lines of knowledge to perfection. He sees clearly what course his research should take and writes grants that his peers are willing to fund.
There is some truth to this, again, depending on the specific grant mechanism. For example, the flagship grant of the NIH, the largest grant awarded to individual investigator (the R01), tends to emphasize research that is well-supported by preliminary data. One reason, of course, is that R01 grants can span up to five years and between $1 and $2 million. If such large investements are made in projects that are too speculative, the vast majority of them will fail and most of the money will have been wasted. Of course, the other tendency is to invest in science that is very well supported by preliminary data and therefore “safe,” meaning that the project proposed will almost certainly yield results. The problem with this latter approach is, of course, that it is so safe. Such projects are unlikely to challenge existing paradigms or create whole new areas of inquiry. Obviously, there needs to be a balance between risky, speculative projects (that are nonetheless well-supported scientifically) that could hit a home run if they succeed and “safe” projects that will produce incremental gains in knowledge and improvements in therapy (i.e., are evolutionary, rather than revolutionary, the equivalent of a base hit instead of a home run).
Where the pendulum swings between these two extremes depends upon funding levels. When funding gets tight, as it has been the last few years, reviewers tend to be more reluctant to fund riskier research, even if it is very compelling and interesting science, because they do not want to throw money at projects with a low chance of success. The “whiner” in me can’t resist pointing out that one way to get more “risky” science funded is to increase science funding overall, to make reviewers more willing to take risks, but that would just reveal me to be a tool of the system, just as Mark Crislip is a shill for big pharma and a tool of the medical-industrial complex. When the pay line (the percentile cutoff for funding) is the 20th percentile rather than the 10th percentile, reviewers are a bit more willing to score risky but brilliant projects higher and cut the applicants some slack for not having as much preliminary data as they normally might like to see. When the payline starts to plunge below the 10th percentile, scientific conservatism reigns.
In any case, there are other grant mechanisms, such as the R21, which provide smaller grants for shorter periods of time (usually one to two years) for riskier projects. Moreover, over the last few years, there has been a real push within the NIH to change the reviewing criteria for grants to emphasize novelty and impact more. Even so, if a tendency towards conservative science was Miller’s main critique of the system, I’d probably agree for the most part, but I’d also point out that the NIH is actually trying to remedy that situation. Unfortunately, Miller couldn’t resist tipping his hand that real reform of the system is not what his polemic was about. After listing what he characterized as “state-sanctioned unassailable paradigms” that will never be funded, Miller devolved completely into ranting crankery:
The human-caused global warming paradigm is most likely false (Soon et al., 2001; Editorial, 2006). Two climate astrophysicists, Willie Soon and Sallie Baliunas, present evidence that shows the climate of the 20th century fell within the range experienced during the past 1,000 years. Compared with other centuries, it was not unusual (Soon and Baliunas, 2003). Unable to obtain grants from NASA (National Aeronautics and Space Administration), Soon (personal communication, August 31, 2006) observes that NASA funds programs mainly on social-political reasoning rather than science.
Duesberg (1996), Hodgkinson (2003), Lang (1993-2005), Liversidge (2001/2002), Maggiore (2000), and Miller (2006), among others, have questioned the germ theory of AIDS. All 30 diseases (which include an asymptomatic low T-cell count) in the syndrome called AIDS existed before HIV was discovered and still occur without antibodies to this virus being present. At a press conference in 1984 government officials announced that a newly discovered retrovirus, HIV, is the probable cause of AIDS, which at that time numbered 12 diseases (Duesberg, 1995, p. 5). Soon thereafter “HIV causes AIDS” achieved paradigm status. But, beginning with Peter Duesberg, Professor of Molecular and Cell Biology at the University of California, Berkeley, a growing number of scientists, physicians, investigative journalists, and HIV positive people have concluded that HIV/AIDS is a false paradigm. The NIH awarded Duesberg a long-term Outstanding Investigator Grant and a Fogarty fellowship to spend a year on the NIH campus studying cancer genes, and he was nominated for a Nobel Prize. When Duesberg publicly rejected the HIV/AIDS paradigm the NIH and other funding agencies ceased awarding him grants. Government-appointed peer reviewers have rejected his last 24 grant applications. Peter Duesberg (personal communication, September 20, 2006) writes: “When I was the blue-eyed boy finding oncogenes and ‘deadly’ viruses, I was 100% fundable. Since I questioned the HIV-AIDS hypothesis of the NIH’s Dr. Gallo, and then the cancer-oncogene hypothesis of Bishop-Varmus- Weinberg-Vogelstein etc. I became 100% unfundable. I was transformed from a virus- and cancer-chasing Angel to ‘Lucifer.’”
Yes, anthropogenic global warming denialism and HIV/AIDS denialism (coupled with the oft-repeated cry of “martyrdom!” from Peter Duesberg, yet!) are what Dr. Miller is about. (Hint to Dr. Miller: Citing Christine Maggiore and Peter Duesberg is not a particularly good way to bolster the credibility of your arguments.) Other “unassailable paradigms” that Miller lists are not quite as ridiculous as his examples of AIDS and global warming, but they’re mostly strawmen; for example, the claim that “cholesterol and saturated fats cause coronary artery disease” is actually not quite what medical science states; rather it is that cholesterol and saturated fats are major factors, among others, that contribute to the pathogenesis of coronary artery disease. (Perhaps Dr. Miller is a member of The International Network of Cholesterol Skeptics, a.k.a. THINCS, which Harriet Hall so delightfully skewered last year.) Using these examples did not exactly bolster Miller’s credibility or case, either. Miller then went on a tear about how science is in service of the state, pulling out more HIV/AIDS denialism coupled with some rather blatant conspiracy-mongering:
AIDS research serves the interest of the state by focusing on HIV as an equal opportunity cause of AIDS. This infectious, egalitarian cause exempts the two primary AIDS risk groups, gay men and intravenous drug users, from any blame in acquiring the disease(s) owing to their behavioral choices. Duesberg, Koehnlein, and Rasnick (2003) hypothesize that AIDS is caused by three other things, singly or in combination, rather than HIV: 1) long-term, heavy-duty recreational drug use–cocaine, amphetamines, heroin, and nitrite inhalants; 2) antiretroviral drugs doctors prescribe to people who are HIV positive– DNA chain terminators, like AZT, and protease inhibitors; and 3) malnutrition and bad water, which is the cause of “AIDS” in Africa. HIV/AIDS has become a multibillion dollar enterprise on an international level. Government, industry, and medical vested interests protect the HIV/AIDS paradigm. The government-controlled peer review grant system is a key tool for protecting paradigms like this.
Personally, I always marvel at such amazing nonsense. For one thing, why would it “serve the interest of the state” for HIV/AIDS to be a “equal opportunity cause of AIDS.” Surely, it could equally serve the interest of the state to demonize one or two unpopular segments of the population, depending upon who is in charge at any one given time. Be that as it may, the evidence that HIV causes AIDS is exceedingly strong and has not been seriously challenged, not by Duesberg, and certainly not by any of Dr. Miller’s HIV “dissident” tracts published at the execrable LewRockwell.com. The views in his articles alone show that Dr. Miller’s critical thinking skills leave much to be desired, and this lack of critical thinking is very apparent in his article attacking peer review.
No wonder Dr. Miller was so unhappy about how peer review works! No wonder his article is such a gem of crankery that I remembered it two years later and have been meaning to discuss it for SBM almost since SBM began.
Personally, my view is that, whatever problems are inherent in the current peer review system (and, as I have said before, there are many), one thing the current system does do a reasonably good job of is of keeping pseudoscience (such as what Dr. Miller apparently subscribes to) from being funded by government bodies. I know, I know. There is one glaring–and I do mean glaring–exception to this rule. You probably know what it is. That’s right; I’m referring to the National Center for Complementary and Alternative Medicine (NCCAM), which exists primarily to fund pseudoscience. True, it does fund studies of herbal remedies, which could be considered a part of pharmacognosy, but such studies would be much better if done by one of the other relevant Institutes or Centers that aren’t dedicated to woo. As for the rest of the “alternative” medicines of which NCCAM funds studies, the vast majority are highly implausible, and NCCAM has utterly failed to produce a single truly positive study or to conclude that any woo doesn’t work. That’s why I have castigated NCCAM time and time again, and, with other SBM bloggers, called for President Obama to defund this agency. Indeed, I even suggested a how-to guide on defunding NCCAM.
Somehow, though, I doubt that NCCAM is what Dr. Miller had in mind when he castigated peer review by government granting agencies. To me, its past ability to keep pseudoscience from being funded and, for the most part, published was one of the great strengths of our peer review system. Unfortunately, the funding of NCCAM shows just how fragile that ability can be. Any reform that is undertaken must be done carefully in such a way as to minimize any weakening this firewall against ideas that are clearly without scientific merit and overwhelmingly believed to be so by scientists. After all, one of the risks of funding “riskier” science is that pseudoscience will sneak in, along with the legitimate science. Worse, one of the risks of funding based far more on ideology than science, which is what NCCAM does given that it was imposed upon the NIH by powerful quackery-friendly legislators like Tom Harkin (who was most unhappy earlier this year because NCCAM was too scientific for his taste in that it hadn’t “proven” more alternative medical modalities to be effective) rather than developed by scientists to fill a perceived need in response to a groundswell of support, is that more pseudoscience will be funded. Of course, that’s exactly what Dr. Miller wants: Legitimacy and funding for the pseudoscience behind HIV/AIDS denialism. Finally, one thing that I have to wonder about is this: If the “unassailable state-sanctioned paradigms” that Dr. Miller detests so much are, as he seems to believe, due primarily to the inherent bias of the NIH grant peer review system that is “designed to protect the existing paradigm,” why, then, is it that scientists around the world also consider Duesberg’s ideas about HIV to be profoundly incorrect and have come, after much wrangling, to believe that human-caused global warming is occurring?
Sadly, the ideas for reform seen in Miller’s article and elsewhere among the HIV/AIDS “dissidents” seem to boil down to either “let’s find a way to fund potential cranks like us” (a.k.a. “mandatory funding of contrarian research”) or “let’s get rid of peer review.” Dr. Miller opines:
One alternative to the competitive peer review grant system that the NIH and NSF might consider for funding specific research projects is DARPA, the Defense Advance Research Projects Agency. This agency manages and directs selected research for the Department of Defense. At least up until now it has been “an entrepreneurial technical organization unfettered by tradition or conventional thinking” within one of the world’s most entrenched bureaucracies (Van Atta et al., 2003). Eighty project managers, who each handle $10-50 million, are given free reign to foster advanced technologies and systems that create “revolutionary” advantages for the U.S. military. Managers, not subject to peer review or top-down management, provide grants to investigators whom they think can challenge existing approaches to fighting wars. As long as the state controls funding for research, managers like this might help break the logjam of innovation in the biomedical and physical sciences. Science under the government grant system has failed and new kinds of funding, with less government control, are sorely needed.
I fail to see how giving appointed managers this power would be “less” government control over research. After all, who hires these managers? The government! What’s to stop the government “orthodoxy” from simply hiring managers who believe the “government orthodoxy” and will therefore disburse research funds in order to support the “orthodoxy”? Nothing! After all, it would be even easier to enforce an orthodoxy if the managers, rather than largely volunteer peer reviewers drawn from diverse academic settings, controlled funding. Besides, military technology, although a broad area, is applied, not basic, science. It probably does not require as much of an understanding of the nitty-gritty of the basic science behind technology proposals as it does to understand whether a basic science or translational research proposal is reasonable, innovative, and feasible. Moreover, remember that the entire yearly budget of the NIH is only around $30 billion, and the entire budget of the NCI is less than $5 billion, both of which are utterly dwarfed by the size of the Defense budget. In other words, the military is much more lavishly appointed and can afford to throw money at risky scientific projects in a way that the NIH and NSF cannot. Moreover, contrary to this example of DARPA, the system that the U.S. Department of Defense uses to evaluate most submitted research proposals is actually peer review. Using peer review, in fact, the Army (believe it or not!) does quite a good job of emphasizing and fostering innovative proposals; indeed, for its biomedical research programs, the Army probably does better job in many respects of supporting scientific innovation than the NIH. If the NIH is going to emulate the military, it would do far better to examine how the Army conducts its scientific peer review sessions, rather than to listen to the posturings of people like Miller.
The rest of the peer review bashers tend not to do much better than Dr. Miller. For example, going back in time a couple of years, one of the most vociferous critics of peer review that I’ve ever encountered (not to mention a hardcore HIV/AIDS denialist) is blogger Dean Esmay. I refer to his ideas, even though he expressed them a couple of years ago, because he was quite impressed with Dr. Miller’s article and decided to add his ideas to Miller’s. Basically, Dean’s ideas (old version of post in the archive) seem to boil down to the sort of reasonable to the ignorant to the unworkable. For example, Dean proposed a seemingly not entirely unreasonable idea of completely eliminating the anonymity of peer reviewers that betrays his ignorance of the process. For one thing, he doesn’t seem to have noticed that Study Section rosters are already published on the web, allowing reasonable guesses as to who specific reviewers are for applications. (In fact, the NIH helpfully sends applicants the complete roster of the members of the study section that reviewed their grant, along with the summary statements and reviews.) He also seems not to understand that it is a not infrequent occurrence for more junior faculty to be reviewing applications by senior, well-entrenched faculty, the veritable “gods” of the field, if you will. How willing would these early mid-career scientists be to provide brutally honest feedback about a bad proposal if the applicant would know who gave him the bad score? In fact, I have felt that very pressure myself. After reading a truly execrable grant application by a very highly regarded scientist that looked as though it had been thrown together over a weekend, I’m not sure I would have had the courage to give it the review it deserved if I knew that the applicant would know who I was. Indeed, completely eliminating anonymity might actually have the tendency to worsen the very problem Dean and Miller decry by leading to grants by highly established and respected scientists getting even more of a pass from study sections than they do already.
In addition, Dean proposed another idea that revealed his ignorance of the NIH, namely to make peer review funding boards “truly multidisciplinary” (whatever that means) and forcing every application to be looked at by a mathematician or someone with a “background in mathematics.” I’m not sure if he was referring to the study sections, which do initial peer review, or advisory councils of each institute, which do the second tier of peer review taking into account specific scientific and/or programmatic priorities of their Institutes, but Dean apparently has never actually looked at the roster of a few typical NIH Study Sections. If he meant study sections, I point out that they already are multidisciplinary, and virtually all of them include biostatisticians! (I hope that’s “mathematical” enough for Dean.) For example, the study section on which I sat until last year included internists, physiologists, surgeons, computer experts, radiologists, medical imaging experts, molecular biologists, a medical physicist, and biostatisticians. I suppose that we could make things even more “multidisciplinary” with “no direct interest” in the field (we could bring in an archaeologist, I suppose, to look at cancer biology proposals), but we would do so at the risk of decreasing the familiarity of reviewers and study section members with the detailed science behind grant applications assigned to them. (On second thought, maybe that’s just what Dean would like. On third thought, there’s no “maybe” about it.) On the other hand, if Dean meant Institute advisory councils, it is hard to see what added benefit that making the these second tier reviewers even more “multidisciplinary” would provide, given that the primary driver of what gets funded is the review provided by the study section, not the post-review committees, which largely rely on the study section’s priority score and the priorities of their respective Institutes to dole out funds. They tend not to make a big difference except for close calls or in cases of proposals that are highly congruent with the Institute priorities but missed the funding payline by a relatively small margin. Making these advisory councils more “multidisciplinary” would be unlikely to affect these priorities because it is not the advisory councils who determine NIH funding priorities; they only implement them. It is the NIH Director and the Directors of the various Institutes, who are appointed by the President, who determine NIH funding priorities, heavily influenced, of course, by Congress and the President.
No one denies that there are problems with the NIH peer review system for grant evaluation; like all human endeavors, there’s room for improvement. Indeed, the complaints bubbling up against it over the last few years are nothing new; I heard the same complaints when I was in graduate school in the early 1990s. Ironically enough, that, too, was a time of very tight paylines, even tighter than they are now, which makes me wonder if the recent budget travails of the NIH are a blessing in disguise, in that they are catalyzing a discussion of how we as a nation can do better with peer review and thus better at deciding what grants to fund. I’ll grant you that the recent ARRA Challenge Grant fiasco, in which we have 20,000 applications chasing between 200-400 Challenge Grants actually isn’t the best argument for this. However, that’s not because of peer reviewers; rather it’s because of a policy decision to put so little of the ARRA stimulus funds into new grants of this sort, which has led to a shortage of reviewers and likely the need to review grants so fast that peer review may suffer. Fortunately, this is a one-time program.
Despite those problems, the system has largely served us well for the last several decades, as long as politicians are kept as much as possible out of the mix, as what failed to happen with NCCAM. Despite their flaws, the NIH and NSF peer review systems, once in place, have remained remarkably immune to political influence and corruption, at least as much as any government entity can be. Certainly they have much to recommend them. For example, junior scientists compete for funds with more senior scientists on a more equal footing than perhaps any other nation in the world. In fact, new investigators who have never received a major NIH award are even given a significant (although, some would argue, not significant enough) break on funding lines to give them a better chance of being funded. Also, applicants can propose virtually any sort of health science-related research project, and it will be seriously considered for grant funding by a study section composed of experts qualified to evaluate it. Moreover, scientists are actively working to address the system’s shortcomings. Meanwhile, contrary to the impression given by Dr. Miller’s article of a system that scientists accept and never challenge, articles about the problems in the peer review system, and there has been much discussion of this at meetings that I have attended, and the NIH even maintains a web page called Enhancing Peer Review.
Of course, substantive and real reform of the peer review system in order to make it function better and allow the funding of meritorious but risky projects is not the true goal of “critics” like Miller, Esmay, and others. Neutering it is, the better to allow pseudoscience like HIV/AIDS denialism an opening. Mark Hoofnagle was right two years ago to warn us to beware the bashers of peer review.