Reading Medical Literature with a Critical Eye

A long time ago I read a study about what makes a good doctor. Some things you might think were important, like grades in medical school, were irrelevant. What correlated the best was the number of medical journals a doctor read. I don’t know whether that means good doctors read more journals or reading more journals makes a better doctor.

One thing I do know is that most of us could learn better journal-reading skills. When I was a busy clinician, I did what I suspect many busy clinicians do: I let the journals pile up for a while, then tackled a stack when I got motivated. I would skim the table of contents to pick out articles that I wanted to read, then I would read the abstracts of those articles. If the abstract interested me, I would read the discussion section of the article. If I was still interested, I might go back and read the entire article. But until after I retired, I never really developed the skills to evaluate the quality of the study.

I knew enough not to jump on the bandwagon the first time something was reported, because I had seen promising treatments bite the dust with further testing. But I really wasn’t aware of all the things that can go wrong in a study, and I didn’t know what to look for to decide if the results were really credible. I’m not an academic; I thought the authors knew a lot more than I did, and I trusted them to a degree that was not warranted.

Eventually I developed some critical thinking skills. I’m still learning. I think what taught me the most was reading appallingly bad studies (thanks to supplement manufacturers, chiropractors, energy medicine proponents, and others!). Once I was aware of bad practices, I knew to look for signs of them even in relatively good studies.

I learned a lot from an excellent book, Critical Thinking About Research by Julian Meltzoff. While directed at psychology, its lessons pertain to research in any field. It has chapters covering all the main aspects of research, like sample selection and controlling for confounding variables, but the best part of the book is a series of 16 practice articles. These are made-up studies with flaws deliberately implanted. You get a chance to look for the flaws, then to check your answers against Meltzoff’s comments. An extra added attraction is the puns in the names of the studies’ authors. For instance, the authors of a study on the social effects of tax deadlines are “Levy” and “Hertz.” And Meltzoff even explains the puns for those who don’t get them.

I also learned a lot from hearing other doctors critique studies. Now there is another great opportunity to benefit from that kind of experience. The American Academy of Family Physicians (AAFP) has initiated a monthly “Journal Club” series in its flagship journal American Family Physician.

Each month, three presenters will review an interesting journal article in a conversational manner. These articles will involve “hot topics” that affect family physicians or will “bust” commonly held medical myths. The presenters will give their opinions about the clinical value of the studies discussed. The opinions reflect the views of the presenters, not those of AFP or the AAFP.

In the April installment, they asked “Does the widespread use of the thrombolytic tissue plasminogen activator (t-PA) produce more benefit or harm in patients who experience an acute stroke?” and discussed a recent review article.

There are 2 kinds of stroke, ischemic and hemorrhagic. Either a part of the brain is deprived of blood (usually from blockage by a clot), or there is bleeding into a part of the brain. If there is a clot, t-PA can be administered to dissolve the clot. The public has learned it is important to rush to the hospital when stroke symptoms begin, because there is only a 3 hour window for using t-PA for an ischemic stroke. Unfortunately, t-PA can cause bleeding complications and might even precipitate the other kind of stroke. The review article showed that among 248,964 patients with ischemic stroke, 1% of them received t-PA. Those who received it had a mortality of 11.4%; those who didn’t had a mortality of 6.8%. What? Yes, patients were more likely to die with t-PA treatment than without.

The discussers ask whether perhaps patients with more severe strokes were likely to get t-PA, whether there was good compliance with the intricate protocol, or whether t-PA was administered to patients who had a “stroke mimic.”

There has only been one good randomized trial.

At three months, 50 percent of patients who received t-PA had minimal or no disability compared with 38 percent who received placebo. This 12 percent difference translates into an NNT [number needed to treat] of eight. [You have to treat 8 patients for 1 to benefit,] In the NINDS trial, there was no increase in mortality rates, but the rate of intracerebral hemorrhage was 6.4 percent in patients receiving t-PA and 0.6 percent in patients receiving placebo (NNH [number needed to harm] = 17).

The bottom line is that one in eight patients is helped at three months, one in 17 is harmed, and although the randomized trial showed no increase in mortality, there has been a documented increase in death rates in patients who have received t-PA therapy outside of research trials. One of the discussers said, “When I am asked what I would do if it was my own family member, I answer honestly: I would not give this therapy.”

Among their main teaching points:

(1) The use of t-PA for acute ischemic stroke is a double-edged sword – both benefit and deleterious effects are noted.
(2) Informed consent, in language that the patient and his or her family can understand, is absolutely necessary when contemplating the use of t-PA for acute ischemic stroke.
(3) The demonstrated efficacy of a drug or intervention in a clinical trial may not translate to effectiveness in the community.
(4) NNT and NNH are powerful tools in documenting an intervention’s effect.

The distinction between efficacy and effectiveness is particularly important to understand. I applaud the AAFP for publishing this series. We need much more of this kind of thing.


Posted in: Pharmaceuticals, Science and Medicine

Leave a Comment (5) ↓

5 thoughts on “Reading Medical Literature with a Critical Eye

  1. overshoot says:

    WRT learning to read journal articles:

    A while back a co-worker related that as an student she had been employed by one of her professors to screen journals. She was to read a largish stack each month and perform what we might call “triage” on them: some topics are non-starters, some are directly on the professor’s list of hot topics and get passed along immediately, and the rest she was to read and critique. Those got a one-page summary (never mind the abstract) with ratings on the methods, novelty of approach, etc.

    She said that she learned more from that one job than from everything else that year, and I suspect that something similar would be beneficial in med school, although given the scheduling issues it might be challenging to work “journal reading” into the curriculum. It might be necessary to cut out something else, like acupuncture.

  2. delaneypa says:

    Good points, I try to read once article a day. But with 1500 articles published daily in the medical literature it’s impossible to process a measureable fraction of the information by using original articles in clinic practice.

    As also mentioned above, much of the medical literature is crap, written by people who do “research” for a variety of reasons….to finish their degree, to add a line to their CV, to use it for marketing purposes. Fortunately, there is enough high quality work out there by dedicated researchers that even their tiny fraction amounts to a large body of quality research.

    Typically what primary care doctors such as myself read are not the original studies. Instead they are usually meta-analyses, systematic reviews, commentary about original studies, textbooks…..pre-digested material of all sorts. I will refer to original articles when very compelling studies mandate immediate changes in practice (e.g., Avandia), or am presenting material to colleagues.

    While this makes my life easier, I sometimes wonder if listening to a Podcast about a commentary about a meta-analysis about original articles can be trusted to propagate the original data accurately.

  3. daedalus2u says:

    There is absolutely no substitute for quantity in getting to understand the literature, what is a good paper and what isn’t. You have to read lots of good papers but also lots of bad papers so that you can recognize which is which.

    But what is “good” depends on what you are going to the literature for. For actual treatments of actual patients, reliability, evaluation of that reliability by peers and traceability of that evaluation of reliability by peers is much more important than is immediacy or originality. Or that is my opinion, since I don’t do any actual treatment.

    When I do a literature survey, my preference is to get everything that I can, the good, the bad, and even the ugly. Once I have it in my files I can evaluate it at my leisure. Having a bad paper that quacks cite is important to be able to discuss just how bad it is, for what reasons and why.

    My main goal in most of the literature research I do, is to find out how nitric oxide fits with that particular physiological pathway I am interested in, and to figure out how raising NO levels would perturb that physiological pathway. If NO is an actual signaling molecule inside the feedback loop that regulates that particular pathway, then it is easy to figure out how low basal NO will perturb it.

    For example bone stiffness is regulated by bone strain causing NO release, that NO activates osteoblasts which deposit more mineral in the regions where the strain is the highest, where the NO level is the highest. Low NO shifts the operating point of the bone strain/NO release control loop to reduced bone stiffness and so causes osteoporosis. Most all of the diseases that are associated with low NO cause osteoporosis and vice versa.

  4. durvit says:

    A while ago I was at a gathering with doctors who are considerably older than Dr Hall and trained in the UK. The issue of keeping up with the journals came up in general conversation. By and large they remarked that they had trained during WWII when their ability to use a stirrup-pump (for fire-fighting rather than medical purposes) and to deal with battlefield injuries was more important than other medicine available at that time. Others trained at the point of social optimism that the introduction of the NHS would transform the health of the nation within a generation and that the notion of significant future medical developments that they would need to know about was almost outlandish (social engineering, town planning and architecture was where people would make the most difference to people’s lives).

    In the post-WWII years they quickly ran into needing to know about the Streptomycin work (done at Cambridge) to deal with the widespread foe of TB. And then they discovered the upside and downside of being able to prescribe antibiotics. They were horrified at the Thalidomide cases and what this told them about the need for sophisticated testing.

    Then, they threw parties at the advent of vaccines for childhood preventable illnesses. But somewhere along the way, they were shocked by Wilson’s assertions about the DTP DPT vaccine (Brian Deer gives a long-term account of this). They were later horrified by Wakefield’s assertions about the MMR and glad that they were mostly retired.

    But along they way, they had seen organ transplants become commonplace and some remarkable innovations. However, the NHS had not abolished the need for itself within a generation – needing only to exist for accidents and similar or congenital disorders.

    Medicine changed around them from having comparatively few drugs to more than you could imagine. They progressed to managing a whole array of illnesses that were discovered and refined as time passed.

    They didn’t learn so much about critical thinking or research literature as students (why would they) and didn’t have ready access to journals when practising. A number of them wondered for how much longer you will be able to study medicine without a prior degree in the UK (during which you would cultivate research skills) but despairing of the burden of debt that that would entail.

Comments are closed.