Articles

Science-Based Medicine 101: How To Establish A Source’s Credibility

I thought I’d do a little SBM 101 series for our lay readers. Forgive me if this information is too basic… at least it’s a place to start for those who are budding scientists and critical thinkers. :)

***

Which news source do you trust more: The New York Times or The National Enquirer? Which news reporter would you trust more: Charlie Gibson or Jerry Springer? As it turns out, medical journals and science researchers run the gamut from highly credible and respected to dishonest and untrustworthy. So as we continue down this road of learning how to evaluate health news, let’s now turn our attention to pillar number one of trustworthy science: credibility.

In medical research, I like to think of credibility in three categories:

1. The credibility of the researcher: does the researcher have a track record of excellence in research methodology? Is he or she well-trained and/or have access to mentors who can shepherd along the research project and avoid the pitfalls of false positive “artifacts?” Has the researcher published previously in highly respected, peer reviewed journals?

2. The credibility of the research: does the study design reflect a clear understanding of potential result confounders and does it control for false positive influences, especially the placebo effect?

3. The credibility of the journal that publishes the research: top tier journals have demonstrated a track record of careful peer review. They have editorial boards of experts who are trained in research methodology and are screened for potential conflicts of interest that could inhibit an objective analysis of the research that they review. The importance of careful peer review must not be underestimated. Some say that the quality of a product is only as good as its quality control system. Top tier journals have the best quality control systems, and the articles they publish must undergo very careful scrutiny before they are published.

So as a lay person, how do you evaluate the credibility of a health news report? In practical terms, here’s what I’d recommend:

1.  Look at the name of the journal reference – where was the research published? Is it from a top tier journal? R. Barker Bausell considers the following journals to be “top tier:” The New England Journal of Medicine (NEJM), The Journal of the American Medical Association (JAMA), Annals of Internal Medicine, Nature, and Science. I might cast a slightly larger net, but no one will argue that these are certainly some of the most respected journals in medicine and science.

2. Look at the study design described in the research article abstract. Was it a randomized, controlled, double-blind, placebo-controlled trial? Were there more than 50 subjects in each group? Did the authors overstate their conclusions? This sort of analysis is challenging to the lay person – so do it if you can, but if it proves too difficult, fall back on credibility check #1.

3. Look at the primary author of the research. Search for his/her name in the National Library of Medicine’s Medline database and see what other research he or she has done, and where it was published.

If the news report is based on credible research, you may feel confident in taking the results more seriously (so long as the media is representing them accurately). But before you hang your hat on a journal’s reputation (they have their own publication biases), let’s take a look at the other 2 pillars of trustworthy science: plausibility and reproducibility. These two will help you navigate your way through the vast gray zone, where the credibility check doesn’t pass with flying colors – or maybe you’re dealing with neither Charlie Gibson nor Jerry Springer.

Posted in: Science and Medicine, Science and the Media

Leave a Comment (14) ↓

14 thoughts on “Science-Based Medicine 101: How To Establish A Source’s Credibility

  1. Gabor Hrasko says:

    It seems the “false positive artifacts?” link is not working.

  2. snfraser says:

    Your last paragraph is instructive. However, the media representation of a study is always questionable. This site is interesting, they evaluate media reporting on health science:

    http://www.mediadoctor.ca/

    But the most important points you make are regarding plausibility and repeatability. These are much more important than your credibility guidelines. I appreciate your cautions with respect to credibility, and I agree a good place to start is with the publication and the researchers track record.

    Linus Pauling is my favorite example to use for my students of where one can go wrong if you only look at the track record. This fellow won 2 Nobel prizes, yet was an advocate for the quack practice of mega-dosing of Vitamin C.

  3. Versus says:

    Here’s another site that grades health news: http://healthnewsreview.org/

  4. tcw says:

    A good post. I’m splitting hairs, but because science is run by fallible humans, the “credibility” criteria has to be taken with a grain of salt. I’m thinking of the Holocaust Museum’s Deadly Medicine book/exhibit, and the classic “Nazi Doctors” book originally published in 86. The credibility of scientists who agreed to the eugenic agenda was ubiquitous at the time, on both sides of the ocean. The problem with the NY Times and C. Gibson is that they may pretend to be unbiased, but that is impossible. I have the feeling that Jerry Springer and the Ntnl. Enquirer at least would admit they are in it for the sophmoric entertainment, not just hiding behond a persona of credibility.

    1. David Gorski says:

      An excellent point. One does have to remember that, back in the latter part of the 19th and first part of the 20th century, racial hygeine was considered a legitimate use of science throughout much of Europe. It’s a cautionary tale, and, unfortunately, at the time it would have been very difficult to recognize racial hygeine as a race-based pseudoscientific abuse of genetics. A number of highly respected scientists totally bought into it and promoted it. Well funded institutes at top universities were set up to study racial hygiene, and this was a couple of decades before the Nazis took power.

  5. bolese22 says:

    Great information for a HS Science teacher to pass onto his students. News media does have its biases and tend to be VERY narrow in what parts of a study they report…whatever will bring them the greatest amount of eyes, rather than a complete explanation in “lay” terms.

  6. EddieVos says:

    About Pauling and “quack practice of mega-dosing of Vitamin C” .. well, he had a better understanding of biochemistry than about anyone ever and there is good reason why H sapiens would benefit from a few grams/day, such as given to zoo primates/per human weight. Pauling worked creatively until his death at age 93 or 94. So much for that anecdote.

    About the evaluation criteria at hand, let’s take another anecdote and see how much the above criteria criteria help, or need improvement: Dr. Rikder with his name on about 1 study/week on Medline so far this year, a Harvard prof and lead author and designer of what was probably a 1/4b$ study published in NEJM 2008 with its design in Circulation 2003. About 18,000 patients randomized to statin or to placebo. Does that not meet all criteria of credibility above? It sure does.

    Now, what about the study results reported? The first study endpoint in Circ. 2003 was a simple “vascular death”, that became “confirmed death from cardiovascular causes” in NEJM 2008 for which the P value has not been officially published [it was probably 0.37] in any of the about 6 study publications so far, while not prior defined or ‘soft’ findings were published with P values.

    This lead author reports a conflict of interest with the Pharma
    financing the study while he and Harvard have patents regarding the study endpoint use leased to the same Pharma.

    Financial conflicts are an issue and if not declared and quantified,
    that issue is more important. For example, the U.S. “when to treat for cholesterol guidelines” were written by people with on average 10 such conflicts, yet they were published in a famous journal, by ‘stellar’ authors.

    Readers therefore should look for what is NOT in a study. Readers should look for NNT’s, Numbers Needed [or Needlessly] Treated for a reasonable time period that is clearly given, and this for INDIVIDUAL clinical disease endpoints, not combined endpoints, and not “surrogate” [lab value] endpoints. Also, look for NNH’s [ditto for Harm] for the side effects.

  7. Amy Romano says:

    This is a nice intro primer, but I think consumers and professionals alike need a more nuanced set of skills. Relying soley or mostly on the credibility of the author or journal is not enough, in my opinion. I’ve reviewed two instances recently of studies that are very misleading but come from the pinnacle of “credible sources”. One recent Cochrane Review misleads the public by reporting that antibiotics are not effective prophylaxis for neonatal mortality from GBS infection, and the 2006 “State of the Science” Conference at NIH (and the AHRQ review underpinning it) comparing so-called “maternal request” cesarean delivery with planned vaginal birth ignored pertinent data and included studies that never should have been included. I could think of so many more examples, even from rigorously peer-reviewed journals and highly respected researchers.

    I would like to second the recommendation of healthnewsreview.org. I wish a site existed that applied the same objective criteria to peer-reviewed research.

  8. snfraser says:

    EddieVos, you have supported my point. Pauling had a great understanding of biochemistry, obviously, yet he was an enormous quack on a particular point, mega-dosing of vitamin C.

    Your other example (as well as Amy Romano’s, whether or not her comments are correct about that particular review, I don’t know, and it doesn’t matter) is precisely why replicability and plausibility are more important than credibility.

    Another way of thinking of credibility is the logical fallacy of appealing to authority. Of course, I agree with the general post that credibility is useful for initially weeding out the quacks and crooks. However, the numerous examples above show this is not the end of the story, as cautioned originally by Val Jones in her post.

  9. EddieVos says:

    Dear snfraser, this may be an ‘academic’ point but Pauling was not a prescribing doctor or someone with patients, i.e. he could not have been a quack the way I understand the term. He was a pure scientist who was wrong in several of the concepts he studied, yet he was right and properly [RCT] supported promoting vitamin C in large amounts to reduce the severity of flu/colds, but not evidently their incidence. He was also right proposing the science of large amounts of B vitamins [we now know] that minimize homocysteine, likely primary cause of vascular disease, Alzheimer’s [Framingham; Nun Study -yet both observational] and much of cancer [read Bruce N. Ames]. As such, Pauling probably did more to promote health and the science thereof than virtually anyone I can think of .. including pharma and excluding the invent of antibiotics.

    Back to the subject of how to judge a credible source of information.
    Much blame lies with our medical journals [I live with them] that don’t
    insist on reporting Numbers Needed to Treat, that allow ‘relative risk
    reductions’ to be promoted almost exclusively [read any statin study,
    statin commentary and especially the meta analysis]. Moreover, many
    journals exist only by the grace of the advertisers and with peer review
    by same-thinking reviewers. Medical editors are having an impossible job too, and many are failing -and some have conflicts of interest themselves!

    Then, there are respectability oozing websites like theheart.org but that ethnically cleanse [sensor] even senior cardiologists not promoting the paradigm du jour but that influence doctor’s thinking. It and its parent/sister sites are FOR PROFIT publications which is not obvious to the lay person.

    Since the lay man cannot realistically weed out the junk and poor
    science, we need doctors who actually read the studies, read the drug
    Prescribing Information in full, and who view with a critical eye the
    about 2000 guidelines that float around for our various diseases. They
    are the ultimate interpreters of what the hard science means for those
    seeking [non quack] advise. Since one American dies every 5 minutes in
    a hospital from the ‘proper’ use of drugs [as per JAMA], much of the
    real damage is done by ‘pharma quacks’ not understanding the science
    themselves. http://www.ncbi.nlm.nih.gov/pubmed/9555760?dopt=Abstract

  10. snfraser says:

    I absolutely agree with your comments about NNT and RRs and failure to note base rates of disease and raw rates etc. (primary training here is in math and stats, btw) in medical journals. Also, interpreting trivial effect sizes in studies that are over powered is a beef of mine, especially without the NNT, NNH etc.

    Anyway, I’ve seen Pauling’s articles in places like Muscle and Fitness magazine promoting mega dosing (1, 2, 3 and more grams) of vitamin C making rather more substantial claims than a reduction in the symptoms of a flu. I just noticed Gorski had a related post: http://www.sciencebasedmedicine.org/?p=236 so I don’t think I need to elaborate further about some of the unsubstantiaed claims by Pauling and the vitamin supplement marketers, who are also out to make a few bucks.

  11. EddieVos says:

    Hi snfraser, thanks pointing to the vitamin C vs cancer blog with a fine science-based last posting/rebuttal. Incidentally, the most productive Randomized Controlled Trial ever was Lind in 1747 using 12 subjects for 6 days and nailing the cure for scurvy.

    The vitamin floggers are no more ethical than those flogging drugs, and Wyeth and Bayer do both and probably make most and massive money at it.

    About vitamin C: once the only producer was Roche at >$30/kg. Then, less than 10 years ago the Chinese came in at $2/1000 grams and the Swiss gave up the business altogether. Now the Chinese with their world and state monopoly have slowly raised their price to about $8/kg. Then again, for 3 grams/day, that would be $8 wholesale for a year, I believe arguably still the best long and short term health bargain [but no cure for cancer].

    Now, about Crestor: once $2.25 for any dose pill before they had data to put against Lipitor with many [all non life saving] studies. Now post JUPITER, arguably THE most misrepresented and over hyped statin study ever, the price jumped by $7 for 30 pills [20 mg] over just the last 2 weeks to $152.ninetynine/30 at RightAid in Newport Vermont [one needs that 'lucky penny' if one is to be among the one in many hundreds to benefit from a single non fatal or soft endpoint: http://www.health-heart.org/JUPITER_Table_3_Outcomes.gif
    The worlds cheapest statin, lovastatin costs $4.99 for 1 year supply [also no life saver] according to the World Health Organization. THAT is worth a ScienceBasedMedicine blog of its own! Ask me.

    Clearly the vitamin and drug hype and industry [and many published studies] are a disgraceful mess … and the masses suffer and pay!

  12. snfraser says:

    Indeed. Take care.

Comments are closed.