Articles

A Skeptical Look at Screening Tests

I’m going to follow Mark Crislip’s example and recycle my presentation from The Amazing Meeting last week, not because I’m lazy or short on time (although I am both), but because I think the information is worth sharing with a larger audience.

We’ve all had screening tests and we’re all likely to have more of them, but there is a lot of misinformation and misunderstanding about what screening tests can and can’t do. Screening tests are done on populations of asymptomatic people and must be distinguished from diagnostic tests done on individual patients who have symptoms. Some tests are excellent for diagnostic purposes but are not appropriate for screening purposes.

We’re constantly being admonished to get tested for one thing or another. A typical example was a recent Dear Abby column. She got a letter from a woman who had been screened for kidney disease and learned that she had a mild decrease in kidney function. Abby was shocked to learn that 26 million Americans have chronic kidney disease, and she advised her readers to get their kidneys checked. This was terrible advice. It superficially seems like good advice, because if you have something wrong with your kidneys, you’d want to know about it, right? In fact, if there was anything wrong anywhere in your body, you’d want to know about it. By that logic, it might seem advisable to test everyone for everything. But that would be stupid. It would find lots of false positives, it would create anxiety by picking up harmless variants and anomalies that never would have caused problems, it would be expensive, and it would do more harm than good.

What makes a good screening test?

Just one example of how screening tests can harm:  If we screened everyone with urine cultures, we would find asymptomatic urinary tract infections in fully 30% of elderly women. Treating those infections provides no benefit. Giving people unnecessary antibiotics would only cause side effects and contribute to antibiotic resistance in bacteria.

So we need to be selective about what tests we use for screening. Here are some of the criteria for a good screening test:

  1. Disease has serious consequences
  2. Screening population has high prevalence of the condition
  3. Not too many false positive or false negative results
  4. Test detects disease before critical point
  5. Test is safe – causes little morbidity
  6. Test is affordable and available
  7. Treatment exists and is not too risky or toxic
  8. Treatment is more effective when started earlier

Sensitivity, specificity, prevalence, and predictive values: a Bayesian analysis

Would this be a good screening test? A press release said “Breaking news! New blood test 87% accurate in detecting Alzheimer’s with an 85% specificity rate. Now we can diagnose it earlier so it can be treated effectively.”

Unfortunately there is no effective treatment for Alzheimer’s, so on that basis alone this wouldn’t be a good screening test. But let’s ignore that for the moment and look at the numbers. What does it mean that a test is 87% accurate and 85% specific?

It means the test is positive in 87% of those who have Alzheimer’s (sensitivity) and negative in 85% of those who don’t have Alzheimer’s (specificity). But those numbers are meaningless unless you know the prevalence of the disease in the population being tested. If you test a population of 10,000 people where the prevalence of Alzheimer’s is 5%, here’s what happens:

Number Positive test Negative test
Have Alzheimer’s 500 87% of 500 = 435 65
Don’t have it 9,500 1,425 85% of 9,500 = 8,075
Total 10,000 1,860 8,140

65 people with Alzheimer’s will have a false negative result and be led to believe they don’t have it when they really do. 1,425 people without Alzheimer’s will have a false positive result and be led to believe they have Alzheimer’s when they really don’t.

If you test positive, what’s the likelihood that you actually have the disease? To get that, you divide the number of true positives (435) by the total number of positives (1,860) and you get 24%. This is the positive predictive value (PPV). If you test negative, what’s the likelihood that you truly don’t have the disease? The number of true negatives (8,075) divided by the total number of negatives (8140) works out to 99%, the negative predictive value (NPV). So this would be a good test for ruling out Alzheimer’s, but it wouldn’t be good for ruling it in. A positive test would only mean there was a 24% chance that you had the disease. The predictive values will be very different for populations with a low prevalence and those with a high prevalence.  Screening tests are much less accurate when the disease is rare.

Mammography

When a woman gets a mammogram, she usually gets a postcard in the mail a few days later. It will say one of two things: “Your mammogram was normal” or “Something looked suspicious; please come back for further evaluation.” If she gets the second postcard, just how worried should she be? Women who get that news tend to panic, but only 10% of them actually have breast cancer. (This percentage is average; it will be higher for women with risk factors. The numbers aren’t important; what’s important is to realize that a positive screening test is not a diagnosis. In most cases, further testing is needed.)

We tend to overestimate the benefits of mammography because of lead time bias, length bias, and over-diagnosis bias. David Gorski has explained those concepts in detail. We know mammography over-diagnoses cancers because there are 22% more diagnoses of invasive cancer in women whose cancers were detected by mammography than in unscreened women who were diagnosed with cancer. That means there must have been cases of invasive cancer in the unscreened group that spontaneously resolved or lost their invasiveness.

Evidence-based recommendations for mammography have recently changed from yearly starting at age 40 to every 2 years starting at age 50. But that’s for the general population. Women at high risk may still be advised to have earlier or more frequent tests.

Mammography saves lives, but it is not as good as most people think.  If 1,000 women are screened yearly for 10 years starting at age 50:

  • 2-10 will be over-diagnosed and treated unnecessarily
  • 5-15 will be diagnosed earlier without any change in the final outcome
  • 500 will have at least one false alarm
  • 250 will undergo biopsy
  • 1 life will be saved
  • 999 would have lived just as long if they had never had a mammogram

(Important: these numbers are for the population as a whole. The benefits of mammography are greater for women at high risk.)

PSA testing for prostate cancer

Prostate cancer is very common but isn’t always harmful. It is found in 80% of autopsies where the men died of something else. Many more men die with prostate cancer than because of it.

The screening test for prostate cancer is a blood test for prostate-specific antigen (PSA). This is not a yes-or-no test. It must be interpreted in the context of the patient’s age and risk factors and the rate of rise, and any cut-off level is arbitrary and will miss some small percentage of cancers. If the PSA test is positive, the next step is biopsy. Typically, 12 needle biopsies are done, 6 on each side. They find cancer in 25% of patients. But if you go back and do more biopsies, you’ll find cancer in 25% more patients. Theoretically, if you could see every cell in the prostate, you might be able to find a cancer cell or two in almost everyone, most of which would never progress or kill the patient. So you have to decide how many biopsies are reasonable. If you find cancer on a biopsy, the next step is treatment, and treatments for prostate cancer are not benign.

In a large European study, screening resulted in an absolute reduction in deaths from prostate cancer of 7 per 10,000 men screened. We can look at this in terms of number needed to screen (NNS) and number needed to treat (NNT). To prevent one death from prostate cancer, 1,068 men would have to be screened and 48 treated. But here’s the kicker: there was no reduction in all-cause mortality. The overall death rate was the same in the screened group as in the unscreened group.

If a prostate cancer is localized and low grade, it is reasonable to observe the patient and not treat unless he develops signs of progression. A recent study compared surgery to watchful waiting and found no reduction in deaths.  Within 30 days of surgery, 1/5 of the patients had complications including deaths. 2 years after surgery, these long term complications were present:

  • 17% were incontinent
  • 81% had erectile dysfunction
  • 12% had bowel dysfunction

Popular advice has been “Get tested; it could save your life” but current expert advice is “Don’t get tested; it does more harm than good.” (Mainly from impotence and incontinence.) Emotional anecdotes abound. One doctor wrote an article titled “The New York Times Killed My Patient.” His patient refused PSA testing because he had read that it was not recommended; he developed invasive prostate cancer and died. Another doctor wrote about the opposite experience: his patient had insisted on testing. He was diagnosed with low-grade localized cancer, the kind that can be observed without treating. But he couldn’t face living with the knowledge that he was harboring an untreated cancer. He was afraid of surgery and opted for radiation treatment. He developed radiation proctitis and had rectal pain and bleeding for years. He became impotent and lost bladder control. He told his doctor he would rather be dead than live wearing adult diapers.

The American Urological Association initially disagreed with the recommendation not to screen, but they have re-considered and issued these new recommendations:

  • Don’t screen men under 40 or over 70
  • Don’t screen men with a life expectancy of less than 10-15 years
  • Don’t screen men age 40-50 who are at average risk
  • Consider screening men age 55-69 who are at average risk
  • Consider screening high risk men of any age
  • Before any screening, doctor should discuss risks and benefits with patient

The United States Preventive Services Task Force

The USPSTF is an independent group of experts who keep up with the current medical literature and issue recommendations based on the best available evidence. Their recommendations are frequently updated and available online.  They’re not perfect, but they’re the most trustworthy source we have, and I rely on them. Their recommendations are not meant to be a cookbook. Their website explains:

The USPSTF recognizes that skilled clinicians serve their patients by individualizing recommendations…to the specific circumstances, values, and perspectives of the individual patient.

They recommend some screening tests be done on everyone: screening for high blood pressure and obesity. Other recommendations are different for different age, sex, or risk groups:

  • Pap smear (age 21-65, every 3-5 years)
  • Colorectal cancer (choice of tests, age 50-75)
  • Diabetes (only if blood pressure is elevated)
  • Cholesterol (men over 35; women and younger men only if they are at increased risk)

Sometimes the recommendations are complicated. Screening for osteoporosis is recommended for women aged 65 years or older and for younger women whose fracture risk is equal to or greater than that of a 65-year old white woman who has no additional risk factors. Men should not be screened. You almost need a translator to decide whether some of the recommendations apply to you: that’s what your doctor is for.

The USPSTF recommends against doing these tests as routine screening tests because they do more harm than good:

  • Annual chest x-ray
  • TB tine test
  • Scoliosis check
  • EKG
  • Teaching patients to do breast and testicular self-exams

Sometimes they’re not sure, and they say so. For glaucoma, they say there is insufficient evidence to recommend either for or against screening.

A recent study found that routine physical exams don’t reduce mortality. Routine visits to a doctor can still be worthwhile, but the physical exam itself (the laying on of hands, looking in the ears, and applying a stethoscope to the chest) is largely a waste of time in the absence of symptoms.

Questionable screening tests offered to the public for profit

Several companies offer ultrasound screening. They come to town, set up in a YMCA or church, and offer “tests your doctor won’t order”: EKG, echocardiogram, ultrasounds to check for narrowing of the carotid arteries, ultrasounds of the abdominal aorta to check for aneurysms, screening for peripheral artery disease (PAD), etc. They offer glowing testimonials from people who believe they would have had a stroke or ruptured aneurysm if they had not been tested, that this screening saves lives. They don’t divulge the reason your doctor won’t order these tests: they do more harm than good. The only one the USPSTF recommends is abdominal aortic aneurysm (AAA) screening and they recommend it be done only once and only in men age 65-75 who have ever smoked. The companies will gladly test everyone and will repeat the test every year when they come back to town. Their advertising is deceptive and their testing is more likely to harm than help.

Electrodermal testing is done with a biofeedback machine hooked up to a computer. It diagnoses all kinds of bogus conditions and “imbalances” and tells the operator which homeopathic remedies and supplements to sell to the customer. It’s about as useful as a Magic Eight-Ball.

Some chiropractors use surface electromyography to locate your subluxations. The test is useless, particularly since there’s no such thing as a chiropractic subluxation.

Whole body CT scans were popular for a while; they were offered in free-standing CT facilities on request, without a doctor’s orders. They exposed patients to the same amount of radiation as 923 chest x-rays. They found abnormalities in 37-86% of patients, creating anxiety and leading to unnecessary follow-up tests that were sometimes invasive and potentially life-threatening. Fortunately they seem to have gone out of style.

When pharmacies in the UK started offering glucose and cholesterol tests, the number of visits to doctors increased, but only 10% of people with positive tests needed treatment.

Talking20 does 17 tests on a drop of blood and plans to increase this to hundreds or thousands of tests you don’t need.

Direct to consumer genetic testing is offered by several companies. They don’t test your entire genome, only specific SNPs (single nucleotide polymorphisms), and they report probabilities based on imperfect data. They might tell you that people with the same SNP as you are 30% more likely to develop Parkinson’s disease than people with other SNPs. So what? There’s nothing you can do to prevent it. And remember, genetics is not destiny. Having the gene for a disease doesn’t mean you will necessarily get the disease. Gene expression depends on environmental factors, epigenetic factors, and complex interactions with other genes. Genetic testing that promises to give you personalized diet and supplement advice is quackery. Our access to genetic information currently exceeds our understanding of what that information actually means.

The latest wrinkle is uBiome testing. It analyzes the genome of the bacteria that live on you and in you. I don’t know what you’re supposed to do with that information.

Conclusion

Screening tests can be very valuable but they can sometimes do more harm than good. They should be chosen judiciously, and the USPSTF offers sound recommendations based on the latest available evidence. More information is not always a good thing; sometimes it’s better not to know.

Posted in: Cancer, Diagnostic tests & procedures

Leave a Comment (24) ↓

24 thoughts on “A Skeptical Look at Screening Tests

  1. Margaret McCartney’s excellent book The Patient Paradox covers this in detail. I recommend it.

    1. anthro49 says:

      I listen to Dr. McCartney on a BBC podcast (Medical Matters). She is just wonderful! She is a champion of straight-talking and never takes up a position on the fence–as many tv doctors do (not talking about Oz, who doesn’t even count as a doctor anymore). No “what’s the harm” for her, and her knowledge of everything is astounding! Sometimes, when she gets wound up, I can barely follow her, between her heavy Scottish speech and the rapidity of same; thank goodness for backup and replay. :-)

  2. anthro49 says:

    Thank you once again, Dr. Hall, for a straightforward presentation of the facts. I will print this one to take with me to the next mammo. Last time when I mentioned the data for self-exam, the woman expressed major shock and tsk-tsked my “ignorance”.
    Same thing all through the biopsy process–I was never really worried, but the time the staff went on attempting to “soothe” me, would have been better spent updating their policies on testing. I didn’t need that biopsy and they made a good show of frightening me into it.

  3. windriven says:

    “Unfortunately there is no effective treatment for Alzheimer’s, so on that basis alone this wouldn’t be a good screening test. ”

    Having watched three people – one related and two not – descend into Alzheimer’s I can tell you with certainty that if I were diagnosed early enough I would suck start a .45. Alzheimer’s is more than I would ask of my family and friends and something I would never wittingly allow myself.

    “1 life will be saved. 999 would have lived just as long if they had never had a mammogram.”

    1:1000. That would be 1000 lives saved per million or 120,000 lives saved in the US over the next 50 or so years (assuming ~120×10^6 women in the US > 30 years old). I’m really not arguing for more mammography, just pointing out that the same numbers can have different emotional impact depending on how they’re presented.

    “Screening tests are done on populations of asymptomatic people and must be distinguished from diagnostic tests done on individual patients who have symptoms.”

    “Teaching patients to do breast and testicular self-exams” is not recommended.

    This frankly flabbergasts me. Not making young men aware that the appearance of lumps, bumps or rough spots on a testicle needs to be evaluated professionally strikes me as negligent. The same for women and lumps in breast tissue. These are symptoms not curiosities and they deserve the attention of a physician if only to assure the patient that they are benign.

    1. David Gorski says:

      1:1000. That would be 1000 lives saved per million or 120,000 lives saved in the US over the next 50 or so years (assuming ~120×10^6 women in the US > 30 years old).

      Indeed. Even pessimistic estimates suggest that in the US mammography probably saves a few thousand lives per year. It’s the law of big numbers. Small percentages can translate into fairly large numbers when applied to a large population.

      1. Harriet Hall says:

        I hope I didn’t leave the impression that mammography wasn’t worthwhile. It most certainly is, The USPSTF recommends doing it every two years from age 50 to 74 on every woman and doesn’t preclude doing it earlier or more often on women at high risk. I only intended to make the point that the public perception of its ability to save lives is exaggerated and that there are harms to be balanced against the benefits.

    2. Sean Duggan says:

      Flip side of large numbers, given the population you described:
      * 240,000-1,200,000 will be over-diagnosed and treated unnecessarily
      * 600,000-1,800,000 will be diagnosed earlier without any change in the final outcome
      * 60,000,000 will have at least one false alarm
      * 300,000,000 will undergo biopsy

      That’s a lot of people undergoing unnecessary treatment to save those 120,000…

      1. David Gorski says:

        Never said there wasn’t a flip side.

      2. windriven says:

        Indeed, Sean. And it would be interesting to see a purely financial analysis of the cost of routine mammography plus resultant biopsies versus the cost of treating the presumably more advanced cancers that would be missed according to these statistics.

        I haven’t a clue how that analysis would play out. And there are enough variables and assumptions that doing it well would be difficult. But both political sides of health care issues play hardball with emotions. Clear and meaningful numbers might put some of that in perspective.

    3. Beamup says:

      This frankly flabbergasts me. Not making young men aware that the appearance of lumps, bumps or rough spots on a testicle needs to be evaluated professionally strikes me as negligent. The same for women and lumps in breast tissue. These are symptoms not curiosities and they deserve the attention of a physician if only to assure the patient that they are benign.

      I believe this is drawing a distinction between being generally aware of the state of one’s breasts/testicles, which is certainly worthwhile, and teaching a specific methodology/schedule, which as I understand it doesn’t add anything.

    4. Rork says:

      Aren’t those estimates of breast cancer-caused death?
      I thought it had still never been shown that overall mortality was altered by screening mamography, just like Hall wrote of PSA screening in today’s post. I thought folks accepted just the result for breast-cancer-caused death, cause they imagine (perhaps rightly, but we really don’t know), that the women that die earlier as a side-effect of screening aren’t that many. That’s not measurement though.
      So maybe there was a bit of a double-standard here.

    5. Harriet Hall says:

      “Not making young men aware that the appearance of lumps, bumps or rough spots on a testicle needs to be evaluated professionally strikes me as negligent. The same for women and lumps in breast tissue.”

      I agree. Doctors should still tell patients to pay attention to their testicles or breasts (indeed, to all parts of their anatomy) and seek help if they notice any changes. The recommendation was only against formal teaching of BSE and testicular self-exam routines. I spent many hours teaching self-exam procedures, and now I realize that the only valuable part of what I said was the last thing I always said, “the details aren’t important; the important thing is to get to know what your breasts/testicles are normally like, and report any changes.” That still stands.

    6. Denise D. says:

      I was personally relieved when the studies were published that found no benefits from breast self-examination. The constant hectoring by advocacy groups smacked of victim-blaming, like all those women who suffered and died may have made it if only their cancers had been found early enough. And if you weren’t diligent about it (like, um, me. And just about all my friends) it was one more thing to feel guilty about.

      And that’s a different matter from actively ignoring signs of cancer (orange peel skin, inflammation, lumps in the breast or armpit, etc). It’s not like we’re supposed to be oblivious to changes in our bodies; it’s just that those elaborate self-examination procedures we were supposed to perform every month turned out to be pointless.

  4. David Gorski says:

    One doctor wrote an article titled “The New York Times Killed My Patient.” His patient refused PSA testing because he had read that it was not recommended; he developed invasive prostate cancer and died.

    Discussed here. :-)

    http://www.sciencebasedmedicine.org/once-more-into-the-screening-breach-the-new-york-times-did-not-kill-your-patient

  5. Sean Duggan says:

    Forwarded on to my Facebook friends (many of whom are more acquaintances, but eh…). We went through the scoliosis screenings in high school after one of my classmates developed severe problems that “should have been detected”. I remember one of my classmates was diagnosed with severe scoliosis with the result turning out to have been the result of them not accounting for increased muscle development on his right side (he was a pitcher for our baseball team). Myself, they put me in the “should follow up” group and then the public outcry for testing died down and I never got evaluated again. I know that my arms are slightly different lengths (not by much, but just enough to make benchpresses awkward if I don’t remember to keep one arm flexed more than the other) but otherwise I’ve stayed in good health.

    I’ve had the glaucoma testing done far too many times, but that’s because I have an inherited issue with deep cupping in my optic nerve that sends most optometrists into a tizzy because it looks like severe glaucoma about to happen. By now, I’ve learned the impending signs and can explain my case, but previously, I had several doctors just say that they “wanted to do an additional test” and next thing I know, I’ve got another line on my bill from insurance and I’m dealing with dilated eyes for the next few hours.

    1. Icewings says:

      Sean, I recently was told by my ophthalmologist that my optic nerve was thin and oval as opposed to thick and round. He asserted that was probably just because of my extreme nearsightedness but recommended a “map” be made of my optic nerve, to set a baseline of its shape. It’s a way to avoid unnecessary tests in the future by glaucoma-fearing docs. Maybe the same thing could be done for you?

    2. Calli Arcale says:

      They don’t dilate your eyes for a glaucoma exam. The glaucoma test (which is part of a routine eye exam) is done by measuring intraocular pressure. A device is gently placed on the eyeball, or squirts a puff of air at the eyeball, while you desperately try not to blink. Dilation of pupils is done to better examine the retina, the complicated light sensing organ at the back of your eye. It is particularly important if you are diabetic or do a lot of sports, as both put you at risk of retina damage. A damaged retina may be manageable, but it is seldom repairable, so it’s important to detect problems as early as possible to have any hope of salvaging some of your vision later in life. There is a new device which allows them to photograph your retina without dilating your pupil, but it costs more than the old method of dripping nightshade in your eyes and then just looking up really close. ;-)

  6. Daniel says:

    Natural remedies and do it yourself solutions are touted as some sort of new age medication now, but nothing can beat the old fashioned way of visiting your doctor and having a proper check up done using the latest medical advancements. It’s sad how so many people refuse to take advantage of that.

  7. Bruce says:

    There are more things that can be done with screening tests than determine treatment. For example, if I knew that I probably would not get Alzheimer’s Disease, I might be more likely to drop my long-term care insurance in the face of large premium increases than if it appeared more likely that I might get it. Being 30% more likely to get Parkinson’s might not lead me to the same conclusion, but better screening tests in the future might help me make this kind of decision. It is true that some screenings are more helpful than others, and good information about what tests actually mean can be extraordinarily helpful. Still, my main point is that treatment decisions are not the only options that can be derived from screening tests. If the data are good, one can use them to make certain kinds of financial and lifestyle decisions as well.

  8. timbartik says:

    Much of the information presented here on PSA screening is misleading. There is over-screening and over-treatment of prostate cancer, but smart screening strategies make sense for many men.

    Here are some things that are misleading in this post’s discussion of prostate cancer screening:

    1. The lack of statistically significant effects of PSA screening on all-cause mortality is meaningless because the European study was never intended to have sufficient sample size and power to detect all-cause mortality effects equal in size to plausible reductions in prostate cancer mortality. All-cause mortality is a very noisy statistic compared to prostate cancer mortality. To have a reasonable power to detect statistically significant all-cause mortality effects similar in size to the prostate cancer mortality reductions in the European study, you would need sample sizes in the millions. We will likely never have such data. Actually, the point estimate in the European study is that all-cause mortality declined at least as much as prostate cancer mortality, but the estimate is simply too noisy to be informative.

    2. The 48 figure presented here is NOT the extra men unnecessarily treated per life saved. Rather, it is the ratio of the extra men with prostate cancer detected in the screening group to the ratio of reduced prostate cancer deaths in the screening group. If you look at the statistical appendices to the European study, you will see that a much higher percentage of the men with prostate cancer in the screening group, compared to the control group, received no treatment other than watchful waiting. This makes sense because their prostate cancers were detected earlier. So, if you calculate the ratio of the extra prostate cancers treated with some treatment other that watchful waiting in the screening group to the prostate cancer deaths averted, you will get a number considerably below 48.

    3. But this calculation as well is misleading because this is the ratio after a certain number of years since screening began, and we know that additional men will be diagnosed and treated for prostate cancer, and possibly die of prostate cancer, as the follow-up period is extended. Researcher Ruth Etzioni and her colleagues have used the European study to project that over a full lifetime, the European study implies that the ratio of additional men treated for prostate cancer due to screening, to the number of prostate cancer deaths averted , is less than 8 to 1. http://journals.lww.com/lww-medicalcare/Abstract/2013/04000/Limitations_of_Basing_Screening_Policies_on.2.aspx

    4. Even the USPSTF does not contend that the empirical evidence shows that prostate cancer treatment leads to an ADDITIONAL 81% of all men being impotent. Remember, this is an older group of men, many of whom would experience some of these problems without any treatment. A more plausible number is that prostate cancer treatment may lead to 50% of all men treated with problems with impotence or incontinence that may be attributed causally to the treatment.

    Now, this all still implies that there may be over-screening and over-treatment for prostate cancer. For men whose expected remaining lifespan is less than 10 years, it seems unlikely that prostate cancer screening or treatment makes sense. And for men who have longer expected lifespans, there are some tough choices here. Screening and treatment for prostate cancer may reduce your risk of death by 10% or so over the next 15 years, but possibly with a 50% risk of very serious side-effects. Different men will value these relative risks differently.

Comments are closed.