I’ve frequently noted that one of the things most detested by quacks and promoters of pseudoscience is peer review. Creationists hate peer review. HIV/AIDS denialists hate it. Anti-vaccine cranks like those at Age of Autism hate it. Indeed, as a friend of mine, Mark Hoofnagle, pointed out several years ago, pseudoscientists and cranks of all stripes hate it. There’s a reason for that, of course, namely that it’s hard to pass peer review if you’re peddling pseudoscience, although, unfortunately, with the rise of “integrative medicine,” it’s nowhere near as difficult as it once was.
Be that as it may, peer review, the process by which scientific papers are evaluated by scientific “peers” to look for problems with the science and decide if the paper is appropriate for publication in a scientific journal, is a concept that dates back hundreds of years. However, for the most part, before the middle of the 20th century, the ultimate determination of whether a paper was appropriate for scientific publication was made by editors or editorial committees. Opinions of external reviewers were sometimes sought when deemed appropriate by journal editors, but by no means was this the practice for most manuscripts. Over the last six or seven decades, external peer review by scientists chosen by the journal editor evaluating a submission has become the standard. Similarly, decisions regarding whether or not to fund grant applications are now generally made by a panel of external reviewers. In the case of the NIH, these panels are called study sections and consist of scientists with expertise in the types of applications being referred to the study section for evaluation, along with (usually) a statistician or two and officials from the NIH who take care of organizing and running the meetings of the panel. The scientific members of a study section usually include “permanent” members, who are assigned to fixed terms on the study section, and ad hoc members, called in for one or a few meetings as needed and deemed necessary by the NIH.