Articles

Science by press release

Last week I wrote about a study that purported to show that antidepressants have no effect in mild to moderate depression. A careful reading of the paper shows that the authors dramatically overstated their findings, particularly in their public statements to the media. The study has another implication beyond the misleading claims about antidepressants. It is an object lesson in an ongoing and disturbing phenomenon in mainstream journalism, the wholesale reprinting of press releases of scientific papers instead of reading and analyzing the papers themselves.

Pick up any newspaper or magazine and you can read about the latest scientific breakthroughs in cancer, Alzheimer’s or heart disease. Just keep in mind that what you are reading is probably a commercial message direct from the authors, not an accurate representation of the paper itself. Medical journalists are supposed to interpret the findings of recent medical publications and present them to the general public in ways that they can understand. They are supposed to provide context for the discovery, explaining what it might mean for disease treatment or cure. Yet, they rarely do. Instead, they simply copy the press release.

Most people are unaware that scientists issue press releases about their work and they are certainly unaware that medical journalists often copy them word for word. Instead of presenting an accurate representation of medical research, medical journalists have become complicit in transmitting inaccurate or deceptive “puff pieces” designed to hype the supposed discovery and hide any deficiencies in the research.

Imagine if a journalist reviewing the newest Ford cross-over vehicle didn’t bother to drive the car, but simply copied the Ford brochure word for word. Could you rely on the journalist’s evaluation? Of course not. Yet that is precisely what medical journalists are doing each and every day.

Not surprisingly, there is a scientific paper describing recent trends in medical press releases. The paper in the of Annals of Internal Medicine, Press Releases by Academic Medical Centers: Not So Academic?, by Woloshin, and colleagues finds:

Of all 113 releases about human studies … [f]orty percent reported on inherently limited studies (for example, sample size <30, uncontrolled interventions, … or unpublished meeting reports). Fewer than half (42%) provided any relevant caveats…

Among the 87 releases about animal or laboratory studies, most (64 of 87) explicitly claimed relevance to human health, yet 90% lacked caveats about extrapolating results to people…

Twenty-nine percent of releases (58 of 200) were rated as exaggerating the finding’s importance…

Almost all releases (195 of 200) included investigator quotes, 26% of which were judged to overstate research importance…

Although 24% (47 of 200) of releases used the word “significant,” only 1 clearly distinguished statistical from clinical significance. All other cases were ambiguous …

Why is this a problem? The harm extends beyond the obvious point that it is deceptive, and a failure of medical journalists to do their job, which is to interpret the accuracy and relevance of scientific publications when writing about them. Because medical journalists credulously publish press release as if they were true, they are constantly publishing conflicting reports, contributing to the public’s distrust of medical research. Each day seems to bring a new report of a food, or a drug that will prevent or cure cancer. Within a week or a month or a year, the journalists are reporting that that food or drug does not prevent or cure cancer.

To the public, it looks like medical researchers are constantly making mistakes. Today they claim that a food will prevent cancer. Next month, the same food will be found to cause cancer. In reality, medical research never demonstrated either claim, but medical journalists reported preliminary findings or flawed research as if they were definitive even though that was untrue.

The willingness of journalists to pass on the information in press releases without checking is not just a function of laziness. Journalists often lack the knowledge of science and statistics that is needed to analyze the paper. Moreover, journalists appear to suffer from a misunderstanding of the scientific literature. Publication of a scientific paper is not the end of a process confirming the truth of a paper; it is only the beginning. Publication does not mean that the findings should be accepted uncritically; it merely means that the findings are worthy of being included in the ongoing public discussion that characterizes science.  The findings of the paper may ultimately be deemed worthless or wrong.

The Annals of Internal Medicine has done an important service in bringing this disturbing practice to light. You can’t believe what you read about medical research in newspapers and magazines because medical journalists are simply copying press releases, not analyzing the research for accuracy or relevance. Therefore, in the interest of accuracy and relevance, I must disclose a caveat to this important scientific paper. In what surely is an unintentional irony, The Annals of Internal Medicine publicly unveiled the paper and its findings by issuing a press release.

Posted in: Science and Medicine, Science and the Media

Leave a Comment (39) ↓

39 thoughts on “Science by press release

  1. Plonit says:

    This phenomenon has been documented regularly in amusing and exhaustive detail at Bad Science http://www.badscience.net/ since 2003. Enjoy.

  2. davidp says:

    Medical journalists are supposed to interpret the findings of recent medical publications and present them to the general public in ways that they can understand.

    Most journalists in most fields stopped doing this long ago – they just reproduce the claims or press releases of interested groups. The journalist’s job is to inexpensively produce words that help fill and sell their media – they are paid by media owners to do that. A few are allowed to do ‘quality investigative journalism’ but that’s rare.

    For extreme press-release ‘journalism’ look at the technology sections of most newspapers.

  3. Thanks for the post. Nice twist at the end, too.

    I’m not sure whether the problem of republishing press releases as content is more common in some disciplines than others, but I know it occurs in education and psychology, too. Perhaps some enterprising student of journalism could systematically compare rates across disciplines.

    Now, on to a guide to recognizing such reports easily. When my suspicions are aroused, I often select a snippet from a report and search for the phrase. When the same phrase appears in more than one source, then I search attributions across the places that published the phrase. Of course, quoting the same material isn’t sufficient evidence of shoddy journalism, but it allows one to check sources and make a judgement about the accuracy and thoroughness of the reporting.

    Then, of course, there’s no substitute for reading the original research!

  4. Scottynuke says:

    Dr. Tuteur;

    Thanks for once again pointing out this problem. As you note, this is not limited to medical journalism. These days, I’m sad to say almost any subject matter requiring serious study will instead get short shrift and the “tell both sides” treatment without objective analysis.

    Journalists are supposed to be something resembling a jack-of-all-trades, able to examine most any issue with a fair degree of skepticism (the right kind). It used to be the case that journalists could eventually focus on a subject area and perhaps become a realiable source of independent analysis. Sadly, that’s only the case today for a select set of major newspapers. I am truly loath to include any TV outlets, even CNN, since they’re so emasculated by the 90-second segment format. *SIGH*

    *starting the stopwatch to see how long it takes Fifi to find fault with this post*

  5. “These days, I’m sad to say almost any subject matter requiring serious study will instead get short shrift and the “tell both sides” treatment without objective analysis.”

    I agree.

    What may be different in science journalism is that writers don’t treat press releases with the same skepticism as they would treat political press releases. I suspect most reporters view political press releases as what they are: spin from a biased source. In contrast, I’m afraid that science journalists view press releases as factual information. Moreover, they don’t realize that publication of a paper does not mean that the paper’s claims have been judge to be true.

    What’s really depressing is that editors of journals are also surprisingly credulous. A substantial portion of what appears in medical journals is junk: the statistical analysis is wrong, the data hasn’t been analyzed at all or the data do not justify the authors conclusions. When editors of journals allow so much bad research into publication it’s not surprising that lay journalists can’t figure out what is true.

  6. anoopbal says:

    I always wondered why journal articles can’t have a section called “Limitations of the study”, just like Method, Discussion, Practical Application and so on.

    This way they have to look at the other side of the coin too and every study have some drawbacks or limitations.

  7. bluedevilRA says:

    I think this press release phenomenon is part of a larger return to yellow journalism. It’s journalistic laziness combined with sensationalism. I stopped using CNN as my homepage because I could not stand the sensationalism coupled with the lack of follow-ups on stories. They would run stories like “Rapid H1N1 flu test inaccurate 90% of the time!” during the H1N1 scare. In the article, they portrayed doctors as relying completely on a rapid test to make a diagnosis. It was ludicrous, not to mention that there is no rapid flu test specifically for H1N1. Eventually, I noticed that the headline changed to “Rapid flu test inaccurate 90% of the time.” However, they made no public comment on the correction (as a good journalist would do).

    This seems to be part of the larger anti-intellectualism movement. “Doctors/experts don’t know what they’re doing.” Such claims are insinuated, yet no attempt is made to actually potray the relevant science (or in the case of the rapid flu test story, the fact that any respectable doctor would know rapid tests to be unreliable and, therefore, never rely on them to make a diagnosis.)

  8. bluedevilRA says:

    scotty, I just laughed out loud at the fifi part of your post. I almost said the same thing. Undoubtedly, someone will find a fault with this post and then it will devolve into ad hom attacks. Dr. Tuteur, you just have a way with people!

    anoop, it is pretty much assumed that in your discussion, you will discuss the limitations of the study. I agree, though, a separate section may be worthwhile just for ease of reading.

  9. edgar says:

    it is rare that 1 30 minute news program does not have 1 piece of medical news. Drives.me.nuts.

  10. Scott says:

    The willingness of journalists to pass on the information in press releases without checking is not just a function of laziness. Journalists often lack the knowledge of science and statistics that is needed to analyze the paper.

    I don’t think this is really fair. The “not just a function of laziness” phrasing carries an (unsupported) implication that it is in part, or even mostly, a function of laziness. It would be better not to (appear to) make such accusations unless you can back it up.

    I don’t doubt that laziness is a factor, at least in some cases, but don’t overstate it.

  11. LovleAnjel says:

    My husband studied to be a journalist but was unable to get a job in his field after college (he graduated just as the intertoobs began to kill off print). He says that science journalists don’t have any training in or knowledge of science beyond what a liberal arts education requires (i.e. a D or better in two gen ed courses). They’re just assigned to the post, as opposed to being assigned the family section or the farm report. Since headline-worthy science tends to be complex, it’s unlikely they are able or even willing to put in the time and effort needed to really analyze the material. I’m betting as more people are laid off and responsibilities increase, this will only get worse.

  12. Scott says:

    @LovleAnjel:

    Depends on the publication, to some extent. The Economist, for instance, prefers people with a science background vs. a journalism background for their science reporters. (According to their advertisements, anyway.) That’s not necessarily typical, but you can definitely see the effects – they’re good at pointing out the limitations of studies they report on and not getting caught up in “tell both sides.”

  13. bluedevilRA says:

    Scott, good point about the Economist. I was unaware that a science background was part of their criteria for science journalists, but it only makes sense. It probably explains why the Economist is one of the few bastions of decent journalism left.

  14. Pat says:

    Some years ago I made a brief study of stories in the Economist dealing with the national economic policy of Malaysia. I hope their science reporters are less biased than their economists. They pretty much made up “scare stories” out of thin air.

  15. mmarsh says:

    I don’t know if medical journals work the same way as computer science journals, but most of the reviews I’ve done have asked the reviewer for a brief summary of the article. While this is a good check for the author(s) that the reviewer understood the paper, it seems like the journal could also make use of this by providing the reviewers’ summaries as alternatives to reading through the paper (trying to figure out what conclusions are actually supported) or relying on the press release.

  16. Zoe237 says:

    Do authors write their own press releases?

    I miss the old intentionally inflammatory Dr. T. ;-)

  17. LovleAnjel says:

    @Scott

    That’s good to know. (I doubt my husband was applying to The Economist, so he probably didn’t see their stricter requirements.)

  18. Wholly Father says:

    Dr Tuteur

    “What’s really depressing is that editors of journals are also surprisingly credulous. A substantial portion of what appears in medical journals is junk: the statistical analysis is wrong, the data hasn’t been analyzed at all or the data do not justify the authors conclusions.”

    I think most journals suffer from a serious shortage of peer reviewers who are qualified to critically analyze the subject matter of a paper AND the statistics. The pool of qualified statisticians within a discipline is small Editors can”t tap into that well for every statistical analysis.

    I am confident that the level of statistical expertise of most journalists is worse than most editors and peer reviewers.

  19. davidp says:

    Shouldn’t we be criticising those who issue “inaccurate or deceptive “puff pieces” designed to hype the supposed discovery and hide any deficiencies in the research”? These are more often university or research institute publicity offices, who should have the time and expertise to

    “interpret the findings of recent medical publications and present them to the general public in ways that they can understand. … to provide context for the discovery, explaining what it might mean for disease treatment or cure”

    Since these publicity offices are part of the institution’s fund raising efforts, issuing inaccurate or deceptive press releases must sometimes amount to extracting funds from the public by dishonesty.

  20. “I think most journals suffer from a serious shortage of peer reviewers who are qualified to critically analyze the subject matter of a paper AND the statistics. The pool of qualified statisticians within a discipline is small Editors can”t tap into that well for every statistical analysis”

    Then they should publish fewer papers. There is no excuse for publishing papers that haven’t been vetted appropriately.

  21. “Shouldn’t we be criticising those who issue “inaccurate or deceptive “puff pieces” designed to hype the supposed discovery and hide any deficiencies in the research”?”

    That would be like criticizing the directors of commercials for deceptive advertising instead of the sponsors.

    There’s no need to publish any press releases. Science is not advanced by press releases. The public does not benefit from press releases. Only those who commission the press releases and those who did the study stand to benefit from press releases.

  22. Scottynuke says:

    Oh, no…

    I have to play Fifi for a moment. :-O (yes, I should be nicer, but six straight days of shoveling after back-to-back blizzards reduces my niceness reserves. And full disclosure — I spent almost 15 years as a reporter and editor, largely on very science-heavy subjects. And I currently write press releases, but not for academia.)

    Dr. Tuteur, the public will indeed benefit from a properly written press release, as long as the writer is truly trying to inform the “person on the street.” Your point that many medical research press releases fail to meet that standard is well taken. If a researcher isn’t comfortable with the plain language of “this is preliminary work, but if we’re on the right track it’s important,” then perhaps they should focus on informing fellow academics instead.

  23. desiree says:

    there actually are “science journalists.” university of california santa cruz and NYU have master’s certificate and degree level programs, to name 2 (and last i knew, there were something like 10 of these in the country). they take bachelors to PhD level scientists and train them in journalism. i graduated from the UCSC program. lots of alumni are in places like new scientist, nature, and in the publicity departments of hospitals, labs, and science associations. it seems that there are fewer in TV and newspapers, but there are still some. still, scientists-turned-journalists are in the vast minority, and tend to concentrate in the more scientific publications.

  24. Scott says:

    If a researcher isn’t comfortable with the plain language of “this is preliminary work, but if we’re on the right track it’s important,” then perhaps they should focus on informing fellow academics instead.

    My objection here is that, for studies where the above is an accurate description, then there is zero point to actually telling the public about it. Only once it’s been established, replicated, and confirmed does it become relevant to the public.

  25. David Gorski says:

    I always wondered why journal articles can’t have a section called “Limitations of the study”, just like Method, Discussion, Practical Application and so on.

    It is almost always required that some verbiage about limitations of the study be included. It goes in the Discussion section, where it’s an important part of the discussion of the study’s findings and their significance. This is particularly true for clinical trials and epidemiological studies. Basic science papers, in my admittedly anecdotal experience, tend to spend less verbiage on their experiments’ limitations.

  26. David Gorski says:

    There’s no need to publish any press releases. Science is not advanced by press releases. The public does not benefit from press releases.

    Although I agree that science is not advanced by press releases, I’m going to have to disagree with your contention that the public categorically does not benefit from press releases. When used judiciously and when they accurately state the findings of studies press releases serve to call attention to the results of worthy research published in peer-reviewed journals that might be of interest to the public. Also, they also serve to inform the public of where their tax dollars are being spent, given that the funding source is almost always listed in press releases and often that source is the NIH or NSF. Finally, in clinical research if a clinical trial is coming about as the result of a study, then a press release can raise awareness in the community and potentially increase the number of subjects recruited.

    I tend to agree that most press releases announcing the results of scientific studies are not just pointless but self-serving, but I think you go too far to conclude that the public categorically does not benefit from press releases. The problem is the abuse of press releases of scientific findings, not press releases themselves.

  27. rosemary says:

    The problem with press releases is when they are passed off as news. This is a very common practice, something I’d guess most businesses in the US do, and it is very common in small local papers as well as TV. The supplement industry and alt medders certainly engage in it along with everyone else. I learned about it years ago when the local ABC network’s 6 o’clock news anchor was doing a “medical breakthrough” segment each night. When I thought “information” given out in one was dangerous, I phoned the station to tell them. The woman I spoke with said, “Oh, we didn’t do that piece. We got it from a syndicate.” However, there was no way for viewers to know that since they never stated it and the anchor did voice overs as video clips were shown of a patient who spoke and appeared to be responding to an interviewer. The impression was that the anchor investigated the story and interviewed those shown when in fact she had never met them or checked out the material presented.

    People in the industry told me that that “is looked down upon and couldn’t happen in a major TV market”. However, when you add up all the people in the small markets, the total number of viewers deceived is really quite large. Industry people also told me about “flaks”, PR specialists who put stories together for clients and get them in the media.

    In small local papers staff reporters and others often interview every new business coming to town writing purely promotional pieces about them which most of the public mistakes for factual information that the reporters and editors have verified.

    I learned all this when I learned about “dietary supplements” and DSHEA and attempted to alert and warn the public. As a result of the discovery, I stopped reading newspapers and magazines and got rid of the TV. I look at Google news just to make sure that I don’t miss major events, but I simply have no faith in most of the media to be bothered with it.

  28. anoopbal says:

    I agree with David. Most of the research never comes out to the public.

    There are a few websites for doctors and health professionals ( example MD consult) which gives journal article summaries and handouts to make better decisions. But the only way the layperson can get a summary of the latest research (done with their tax money) is through press releases. They get the most people so I don’t see a better way.

    We don’t have to throw the baby out of the bathwater. I think what we need is a better way to communicate those research findings. I think there are courses like Health Journalism and Communication for people who are interested in this field.

    But I see where Amy is coming from. With the current model, the press releases are hurting more than they are helping.

  29. anoopbal says:

    Here is a relevant article about communicating medical news in NEJM :

    http://content.nejm.org/cgi/content/full/360/1/1

  30. JMB says:

    Perhaps the press release should be included in the submission of the paper for peer review, so that the press release can be peer reviewed as well.

  31. Fifi says:

    It’s not press releases or even PR that are the root cause of the distortions that get sold to the media. (And blaming the media for not being an ideal it’s never actually been is a bit silly and indulging in unreality-based thinking.)

    Universities exaggerate claims for a reason. It’s because it helps get both private and public funding for the university and research projects. To ignore the publish or perish pressures (meaning both publishing in academic journals and getting general publicity for research) is to ignore reality and the root causes of why and how so much exaggeration goes on. As academic medical science becomes increasingly involved directly involved with patenting and commerce, there’s an even greater motivation to exaggerate claims.

    Drug companies exaggerate claims for a reason – to sell more drugs and make larger profits.

    Media is about selling both newspapers and advertising, people like to read exciting stories so exciting stories get pushed. Massive budget cuts in both specialty and mainstream publications and news outlets over the past decade mean there simply aren’t the resources or specialist journalists to do really good investigative reporting. Why? At least partly because there isn’t public demand (but mainly because publications have always been used to promote publishers’ interests – political, economic and personal). It’s naive to believe otherwise and shows a lack of understanding of how media works.

    Complaining about the effect without actually understanding or contextualizing the cause is just silly and simplistic – and ultimately doesn’t change anything or educate readers.

  32. “Perhaps the press release should be included in the submission of the paper for peer review, so that the press release can be peer reviewed as well.”

    Brilliant! If only we could make that a requirement of publication.

  33. rosemary says:

    fifi, “Complaining about the effect without actually understanding or contextualizing the cause is just silly and simplistic – and ultimately doesn’t change anything or educate readers.”

    If you are addressing my comment above, I think you are wrong. In fact I believe, but am not certain, that passing off press releases promoting products and services as news is a violation of US law. That is based on a very brief discussion with an FTC lawyer at an airport where we were both catching planes after having spoken at the same conference.

    I also know that explaining the situation to individuals, including the board of directors of a non-profit board I am on, has enlightened them and changed their approaches to the media.

    I am very aware of the problems the media has and am surprised that you, a writer, seem to be defending it and not calling it “Big Media”.

    The mainstream media is being massacred by the Internet but before that it greatly overextended itself in an effort to make big profits. Rather than acknowledging that there was no market for more and more of the product it claimed it was selling, news, it changed the product to suit the taste of what more consumers would buy, and it did that to make money. Look at cable TV trying to provide material that mainstream viewers will watch 24/7.

    Regarding science and medical journals, there has been a proliferation of new ones in the recent past which has resulted in a lot of inferior journals and a lot of inferior studies being published. Perhaps that can be blamed on “Big Academia” out to make more money or to get a bigger share of the pie rather than to learn and educate.

    If you make excuses for the media’s dishonesty, I have to conclude that you are not using the same yardstick to judge it as you do for other businesses and industries

  34. anoopbal says:

    “Perhaps the press release should be included in the submission of the paper for peer review, so that the press release can be peer reviewed as well.”

    Brilliant! If only we could make that a requirement of publication.”

    Let’s be a bit realistic here. Considering the logistics involved and how a scientist writes, that’s just wishful thinking.

  35. Fifi says:

    No Rosemary, my post wasn’t directed at anything you wrote. Sure mainstream/popular media is crap and influenced by commercial, political or personal concerns. It almost always has been (going way back before TV). Even when there was actually money put into investigative journalism, it was because breaking juicy and scandalous stories sold newspapers (and often the stories were targeting someone the publisher wasn’t aligned with). People who don’t have a good understanding of publishing and media tend to be taken in by Hollywood stories about the newspaper biz or broadcasting (just like people who aren’t familiar with science think it’s like it is in the movies).

    That doesn’t mean there aren’t highly ethical journalists and editors with a social conscience who really do care about what they’re reporting, of course. (Though few of them still have jobs. You’re quite right that the trend towards wanting newspapers to turn profits – rather than being tools to wield influence – has been bad for print media.) What it means is that people who think it’s like it is in the movies, and then get outraged when they discover popular/mainstream media isn’t like they imagined it to be, are indulging in unreality-based thinking. If you think I sound cynical, you should listen to someone who actually works a news beat!

    However, I don’t expect the general public to be as cynical as someone who works in communication. That said, it’s just extremely naive to believe that mainstream reporting on cars is somehow objective when it’s well known that it’s one of the areas where the most fawning, gee-whiz type reporting and manipulation goes on (along with technology, travel and beauty). Certainly science reporting in MSM has suffered, alongside art, dance and theater, because they’re all specialized areas that aren’t particularly profitable. If you want good quality writing about any of these areas, you really need to buy specialty publications (and actually support the writers and publishers who do care about these subjects). Complaining about popular media for being popular media is a bit like complaining about water for being wet.

  36. Fifi says:

    Of course, critiquing the role the PR departments of universities and pharmaceutical companies play – and medical journals which a reporter should be able to consider an authoritative source – wouldn’t be nearly as sensationalist as being outraged at popular media for doing what popular media has always done. It’s kind of ironic really. But, hey, why bother with context and depth of analysis when one can be shallow and harness sensationalism and emotion instead. See how it works…

  37. biotunes says:

    The other part of this equation is that papers are so badly written that the press-release probably becomes a fall back tool of journalists tired of trying to plow through all the jargon (which is often used to obfuscate bad science):

    http://bioblog.biotunes.org/bioblog/2007/10/04/why-the-public-doesnt-get-science/

  38. Fifi says:

    Great blog post biotunes! And very good point about jargon, which doesn’t only apply to science by an means.

Comments are closed.