Aug 06 2009
Science-Based Medicine 101: How To Establish A Source’s Credibility
I thought I’d do a little SBM 101 series for our lay readers. Forgive me if this information is too basic… at least it’s a place to start for those who are budding scientists and critical thinkers. 
***
Which news source do you trust more: The New York Times or The National Enquirer? Which news reporter would you trust more: Charlie Gibson or Jerry Springer? As it turns out, medical journals and science researchers run the gamut from highly credible and respected to dishonest and untrustworthy. So as we continue down this road of learning how to evaluate health news, let’s now turn our attention to pillar number one of trustworthy science: credibility.
In medical research, I like to think of credibility in three categories:
1. The credibility of the researcher: does the researcher have a track record of excellence in research methodology? Is he or she well-trained and/or have access to mentors who can shepherd along the research project and avoid the pitfalls of false positive “artifacts?” Has the researcher published previously in highly respected, peer reviewed journals?
2. The credibility of the research: does the study design reflect a clear understanding of potential result confounders and does it control for false positive influences, especially the placebo effect?
3. The credibility of the journal that publishes the research: top tier journals have demonstrated a track record of careful peer review. They have editorial boards of experts who are trained in research methodology and are screened for potential conflicts of interest that could inhibit an objective analysis of the research that they review. The importance of careful peer review must not be underestimated. Some say that the quality of a product is only as good as its quality control system. Top tier journals have the best quality control systems, and the articles they publish must undergo very careful scrutiny before they are published.
So as a lay person, how do you evaluate the credibility of a health news report? In practical terms, here’s what I’d recommend:
1. Look at the name of the journal reference – where was the research published? Is it from a top tier journal? R. Barker Bausell considers the following journals to be “top tier:” The New England Journal of Medicine (NEJM), The Journal of the American Medical Association (JAMA), Annals of Internal Medicine, Nature, and Science. I might cast a slightly larger net, but no one will argue that these are certainly some of the most respected journals in medicine and science.
2. Look at the study design described in the research article abstract. Was it a randomized, controlled, double-blind, placebo-controlled trial? Were there more than 50 subjects in each group? Did the authors overstate their conclusions? This sort of analysis is challenging to the lay person – so do it if you can, but if it proves too difficult, fall back on credibility check #1.
3. Look at the primary author of the research. Search for his/her name in the National Library of Medicine’s Medline database and see what other research he or she has done, and where it was published.
If the news report is based on credible research, you may feel confident in taking the results more seriously (so long as the media is representing them accurately). But before you hang your hat on a journal’s reputation (they have their own publication biases), let’s take a look at the other 2 pillars of trustworthy science: plausibility and reproducibility. These two will help you navigate your way through the vast gray zone, where the credibility check doesn’t pass with flying colors – or maybe you’re dealing with neither Charlie Gibson nor Jerry Springer.
14 Responses to “Science-Based Medicine 101: How To Establish A Source’s Credibility”
It seems the “false positive artifacts?” link is not working.
Your last paragraph is instructive. However, the media representation of a study is always questionable. This site is interesting, they evaluate media reporting on health science:
http://www.mediadoctor.ca/
But the most important points you make are regarding plausibility and repeatability. These are much more important than your credibility guidelines. I appreciate your cautions with respect to credibility, and I agree a good place to start is with the publication and the researchers track record.
Linus Pauling is my favorite example to use for my students of where one can go wrong if you only look at the track record. This fellow won 2 Nobel prizes, yet was an advocate for the quack practice of mega-dosing of Vitamin C.
Here’s another site that grades health news: http://healthnewsreview.org/
A good post. I’m splitting hairs, but because science is run by fallible humans, the “credibility” criteria has to be taken with a grain of salt. I’m thinking of the Holocaust Museum’s Deadly Medicine book/exhibit, and the classic “Nazi Doctors” book originally published in 86. The credibility of scientists who agreed to the eugenic agenda was ubiquitous at the time, on both sides of the ocean. The problem with the NY Times and C. Gibson is that they may pretend to be unbiased, but that is impossible. I have the feeling that Jerry Springer and the Ntnl. Enquirer at least would admit they are in it for the sophmoric entertainment, not just hiding behond a persona of credibility.
An excellent point. One does have to remember that, back in the latter part of the 19th and first part of the 20th century, racial hygeine was considered a legitimate use of science throughout much of Europe. It’s a cautionary tale, and, unfortunately, at the time it would have been very difficult to recognize racial hygeine as a race-based pseudoscientific abuse of genetics. A number of highly respected scientists totally bought into it and promoted it. Well funded institutes at top universities were set up to study racial hygiene, and this was a couple of decades before the Nazis took power.
Great information for a HS Science teacher to pass onto his students. News media does have its biases and tend to be VERY narrow in what parts of a study they report…whatever will bring them the greatest amount of eyes, rather than a complete explanation in “lay” terms.
About Pauling and “quack practice of mega-dosing of Vitamin C” .. well, he had a better understanding of biochemistry than about anyone ever and there is good reason why H sapiens would benefit from a few grams/day, such as given to zoo primates/per human weight. Pauling worked creatively until his death at age 93 or 94. So much for that anecdote.
About the evaluation criteria at hand, let’s take another anecdote and see how much the above criteria criteria help, or need improvement: Dr. Rikder with his name on about 1 study/week on Medline so far this year, a Harvard prof and lead author and designer of what was probably a 1/4b$ study published in NEJM 2008 with its design in Circulation 2003. About 18,000 patients randomized to statin or to placebo. Does that not meet all criteria of credibility above? It sure does.
Now, what about the study results reported? The first study endpoint in Circ. 2003 was a simple “vascular death”, that became “confirmed death from cardiovascular causes” in NEJM 2008 for which the P value has not been officially published [it was probably 0.37] in any of the about 6 study publications so far, while not prior defined or ‘soft’ findings were published with P values.
This lead author reports a conflict of interest with the Pharma
financing the study while he and Harvard have patents regarding the study endpoint use leased to the same Pharma.
Financial conflicts are an issue and if not declared and quantified,
that issue is more important. For example, the U.S. “when to treat for cholesterol guidelines” were written by people with on average 10 such conflicts, yet they were published in a famous journal, by ‘stellar’ authors.
Readers therefore should look for what is NOT in a study. Readers should look for NNT’s, Numbers Needed [or Needlessly] Treated for a reasonable time period that is clearly given, and this for INDIVIDUAL clinical disease endpoints, not combined endpoints, and not “surrogate” [lab value] endpoints. Also, look for NNH’s [ditto for Harm] for the side effects.
This is a nice intro primer, but I think consumers and professionals alike need a more nuanced set of skills. Relying soley or mostly on the credibility of the author or journal is not enough, in my opinion. I’ve reviewed two instances recently of studies that are very misleading but come from the pinnacle of “credible sources”. One recent Cochrane Review misleads the public by reporting that antibiotics are not effective prophylaxis for neonatal mortality from GBS infection, and the 2006 “State of the Science” Conference at NIH (and the AHRQ review underpinning it) comparing so-called “maternal request” cesarean delivery with planned vaginal birth ignored pertinent data and included studies that never should have been included. I could think of so many more examples, even from rigorously peer-reviewed journals and highly respected researchers.
I would like to second the recommendation of healthnewsreview.org. I wish a site existed that applied the same objective criteria to peer-reviewed research.
EddieVos, you have supported my point. Pauling had a great understanding of biochemistry, obviously, yet he was an enormous quack on a particular point, mega-dosing of vitamin C.
Your other example (as well as Amy Romano’s, whether or not her comments are correct about that particular review, I don’t know, and it doesn’t matter) is precisely why replicability and plausibility are more important than credibility.
Another way of thinking of credibility is the logical fallacy of appealing to authority. Of course, I agree with the general post that credibility is useful for initially weeding out the quacks and crooks. However, the numerous examples above show this is not the end of the story, as cautioned originally by Val Jones in her post.
Dear snfraser, this may be an ‘academic’ point but Pauling was not a prescribing doctor or someone with patients, i.e. he could not have been a quack the way I understand the term. He was a pure scientist who was wrong in several of the concepts he studied, yet he was right and properly [RCT] supported promoting vitamin C in large amounts to reduce the severity of flu/colds, but not evidently their incidence. He was also right proposing the science of large amounts of B vitamins [we now know] that minimize homocysteine, likely primary cause of vascular disease, Alzheimer’s [Framingham; Nun Study -yet both observational] and much of cancer [read Bruce N. Ames]. As such, Pauling probably did more to promote health and the science thereof than virtually anyone I can think of .. including pharma and excluding the invent of antibiotics.
Back to the subject of how to judge a credible source of information.
Much blame lies with our medical journals [I live with them] that don’t
insist on reporting Numbers Needed to Treat, that allow ‘relative risk
reductions’ to be promoted almost exclusively [read any statin study,
statin commentary and especially the meta analysis]. Moreover, many
journals exist only by the grace of the advertisers and with peer review
by same-thinking reviewers. Medical editors are having an impossible job too, and many are failing -and some have conflicts of interest themselves!
Then, there are respectability oozing websites like theheart.org but that ethnically cleanse [sensor] even senior cardiologists not promoting the paradigm du jour but that influence doctor’s thinking. It and its parent/sister sites are FOR PROFIT publications which is not obvious to the lay person.
Since the lay man cannot realistically weed out the junk and poor
science, we need doctors who actually read the studies, read the drug
Prescribing Information in full, and who view with a critical eye the
about 2000 guidelines that float around for our various diseases. They
are the ultimate interpreters of what the hard science means for those
seeking [non quack] advise. Since one American dies every 5 minutes in
a hospital from the ‘proper’ use of drugs [as per JAMA], much of the
real damage is done by ‘pharma quacks’ not understanding the science
themselves. http://www.ncbi.nlm.nih.gov/pubmed/9555760?dopt=Abstract
I absolutely agree with your comments about NNT and RRs and failure to note base rates of disease and raw rates etc. (primary training here is in math and stats, btw) in medical journals. Also, interpreting trivial effect sizes in studies that are over powered is a beef of mine, especially without the NNT, NNH etc.
Anyway, I’ve seen Pauling’s articles in places like Muscle and Fitness magazine promoting mega dosing (1, 2, 3 and more grams) of vitamin C making rather more substantial claims than a reduction in the symptoms of a flu. I just noticed Gorski had a related post: http://www.sciencebasedmedicine.org/?p=236 so I don’t think I need to elaborate further about some of the unsubstantiaed claims by Pauling and the vitamin supplement marketers, who are also out to make a few bucks.
Hi snfraser, thanks pointing to the vitamin C vs cancer blog with a fine science-based last posting/rebuttal. Incidentally, the most productive Randomized Controlled Trial ever was Lind in 1747 using 12 subjects for 6 days and nailing the cure for scurvy.
The vitamin floggers are no more ethical than those flogging drugs, and Wyeth and Bayer do both and probably make most and massive money at it.
About vitamin C: once the only producer was Roche at >$30/kg. Then, less than 10 years ago the Chinese came in at $2/1000 grams and the Swiss gave up the business altogether. Now the Chinese with their world and state monopoly have slowly raised their price to about $8/kg. Then again, for 3 grams/day, that would be $8 wholesale for a year, I believe arguably still the best long and short term health bargain [but no cure for cancer].
Now, about Crestor: once $2.25 for any dose pill before they had data to put against Lipitor with many [all non life saving] studies. Now post JUPITER, arguably THE most misrepresented and over hyped statin study ever, the price jumped by $7 for 30 pills [20 mg] over just the last 2 weeks to $152.ninetynine/30 at RightAid in Newport Vermont [one needs that 'lucky penny' if one is to be among the one in many hundreds to benefit from a single non fatal or soft endpoint: http://www.health-heart.org/JUPITER_Table_3_Outcomes.gif
The worlds cheapest statin, lovastatin costs $4.99 for 1 year supply [also no life saver] according to the World Health Organization. THAT is worth a ScienceBasedMedicine blog of its own! Ask me.
Clearly the vitamin and drug hype and industry [and many published studies] are a disgraceful mess … and the masses suffer and pay!
Indeed. Take care.
[...] let’s look at what Val Jones, the target of revere’s pedantry, prefaced her post with: I thought I’d do a little SBM 101 series for our lay readers. Forgive me if this information [...]