Reliability of Health Information on the Web

Last week at TAM8 some SBM colleagues (David Gorski, Kimball Atwood, Harriet Hall, Rachel Dunlop) and I gave two workshops on how to find reliable health information on the web. As part of my research for this talk I came across a recent and interesting study that I would like to expand upon further – Quality and Content of Internet-Based Information for Ten Common Orthopaedic Sports Medicine Diagnoses.

The fact that the article focuses on orthopedic diagnoses is probably not relevant to the point of the article itself, which was to assess the accuracy of health information on the web. They looked at 10 orthopedic diagnoses and searched on them using Google and Yahoo, and then chose the top results. They ultimately evaluated 154 different sites with multiple reviewers for quality of content and also for their HON rating.

For background, the HON rating comes from an independent organization, the Heath on the Net Foundation, that rates health care sites on a number of criteria. These include assessment of how authoritative the sources are, the level of transparency, and if opinions expressed are justified with evidence and references. While generally reasonable, the HON assessment does not necessarily involve a thorough assessment of the quality of the science on a given website, and many sites with what I would consider dubious information have earned the HON seal of approval.

Among other things, this new study evaluated how scientifically accurate health information on the websites they reviewed was, and also compared them to their HON ratings. They divided the websites evaluated into various types – non-profit, academic, commercial, and individually run. What they found was that the quality of information was significantly better on non-profit and academic sites than on commercial and individually run sites. This is not surprising – commercial sites are likely to be compromised by a desire to advertise or sell product. But “commercial” also refers to sites that monetize content – not necessarily selling products, but simply providing content as their product in order to sell advertising. This includes sites such as WebMD.

It is also not surprising that individual sites also scored relatively low on average. An individually run site is only as good as the individual running it, so there is bound to be a great deal of variability. Also, individuals are more likely to make mistakes or have missing information than groups.

Non-profit and academic sites are more likely to have editorial policies that emphasize quality and integrity of content. But also they are more likely to have some vetting process for information. At SBM (a non-profit site) for example, we carefully guard our editorial integrity and also provide some layer of editorial oversight.

But the study also provides reason to be cautious, even about the best sites. They rated quality of information on a 100 point scale and found a range of 45-61%. So even the best sites had a mediocre score. This is likely due to the fact that health information is complex and rapidly changing. Nothing short of a thorough editorial and peer-review process is likely to generate both reliable and thorough up-to-date information. This study is therefore reason for all providers of health information on the net to raise their game. There is definitely room for improvement.

The study also found that having the HON seal of approval did significantly correlate with higher quality and integrity scores. So the code does mean something, even if it is still not a guarantee of science-based content.


Further study of health information on the net is warranted as the results of this one study argue for caution. For providers it suggests we need to improve our filters and editorial process to improve the quality of our content. For consumers the results suggest that non-profit and academic health information sites are most reliable, while commercial and individual sites should be viewed with caution.

But further I would suggest to consumers of health information on the net that no single site or article should ever be relied upon for information. The best way to get a thorough and accurate treatment of a health topic is to look at multiple sites. Try to determine what the consensus of opinion and information is, and be very wary of outliers. This is generally good advice for any research, not just health information.

Posted in: Science and the Media

Leave a Comment (18) ↓

18 thoughts on “Reliability of Health Information on the Web

  1. windriven says:

    It would be interesting to understand the criteria used to make these assessments. It seems to me that information meant for the general consumer would necessarily be rather different from information intended for a technical specialist. An entry on ‘sprains’ intended for soccer moms might be accurate and useful for its intended purpose but found incomplete and wanting from the perspective of a sports trainer much less an orthopedic surgeon. Without understanding the scale against which these sites were measured it is difficult to interpret the meaning of the results.

    Caveat emptor. Or is it caveat browser?

  2. aeauooo says:

    Caveat lector

  3. Kevin Swartz says:

    I’m not sure if this would be particularly interesting, but I wonder if the same grading criteria could be applied to review articles or other peer-reviewed resources for comparison to give a better idea of what this 45-61% on their “100 point scale” really means in a relative sense.

  4. qetzal says:

    Dr. Novella,

    Can you comment briefly on what resources practicing MDs turn to? For example, if an MD needed a reliable refresher on any of those orthopedic conditions, where would s/he go? Would they know of readily accessible resources with substantially better accuracy than these websites?

    And if the answer is yes, I wonder why none of the surveyed websites drew on those same resources to ensure better accuracy. I would have thought at least a few of the academic or non-profit sites would try to do so.

  5. aaronupnorth says:

    Qetzal: I too will be interested to see what Dr. Novella says (do neurologists ever need to look things up??).
    Here’s me perspective as an emergency doctor (we are always looking things up.) I have a couple of excellent small paper texts for very specific topics like orthopedic injuries which I leave in my department. The trouble is that the language in them is very jargon heavy so not very accessible. If you don’t know if your patient has a Galleazi or a Monteggia fracture (and seriously, the book I have with the best management advice still uses eponyms) then the book is useless to you.
    For medical conditions I use an online service called uptodate which is generally good quality and the articles are constantly reviewed and updated by experts in each field. The problem is that it is subscription based and about $500 a year (ouch!).
    I, and probably most docs, also use pubmed, or OVIDweb or some other means of directly accessing the medical literature to answer specific questions (though in emerg that is typically after the fact as I rarely have 30 minutes during a shift to sit down and do a proper lit search).
    Most docs will also have subscriptions to the main journals in their area of expertise and be kept abreast of new developments by reading these as well, and sometimes participating in journal clubs where groups of docs discuss the merits of articles (like a really boring book club).
    I think there is some difference between the information docs and patients want on line. Patients want a comprehensive resource that informs them about the condition and their options. I am usually looking for much more specific information; what is the dose for a drug, could this symptom fit with this problem, what is the management for this sort of rare overdose… I also tend to prefer jargon-y material just because it is usually shorter and I am rarely looking for expose on a topic.

  6. “Patients want a comprehensive resource that informs them about the condition and their options. ”

    Also, one thing I find very valuable as a patient is “when to see a doctor” resources. How high a fever, how long, what symptoms are potentially serious and should be checked out immediately (or at least soon) versus ones that you can home treat. I also use recommendations for home treatment when that is advisable.

    Our Health network has a pretty good “Health Advice” website with various topics, including symptoms that covers these questions pretty well. I actually have found it more helpful the the nurse staffed help line that was rather inconsistent in it’s (their?) ‘when to make an appointment’ approach.

  7. Polite Scott and I had and idea for an SBM 201 workshop for TAM9/ SMB3 next year. The topic would be the basics of how to evaluate a study paper/ detailed abstract of a study.

    It would cover basic evaluation of a study from initial hypothesis through conclusions and how to spot red flags showing if the researchers “jumped the shark” at any point in the study and at which point the study might lose credibility. (ie- it’s a good study all the way through until we get to the point that the conclusions aren’t supported by the data vs. the methodology was flawed in the first place.)

    The presenter would give a basic, cursory run through of two or three studies/ detailed abstracts: one fundamentally flawed throughout, one that jumps the shark in the conclusions, and one that is sound end to end. Handouts could be provided for audience members to follow along with and take notes on.

    Also, please bully Peter Lipson into coming next year and get Mark Crislip to come back.

  8. red rabbit says:

    As a family doc, newly minted, I use uptodate as well. It’s a non-trivial expense for a non-trivial service.

    That said, the same authors provide content at emedicine, which requires registration but is free of charge. It’s an excellent site, well-written, with generally up to date and in-depth information, citations of the literature, the whole thing.

    CMAJ (the Canadian Medical Association Journal) is online and you can search the archives. It’s free. They have great EBM reviews.

    Those are the resources I really trust, and everything else gets a jaundiced eye. Re: looking for consensus in the literature: one of the problems on the net sometimes is that (ie with vaccines) the chorus is the large minority of vocal whackjobs. It can be a minefield if you’re starting from relatively little information.

  9. qetzal says:

    As a professional molecular biologist, I’m no stranger to PubMed. I have some pretty fair experience in early stage drug development as well (through Phase II trials), so I think can handle clinical manuscripts fairly well for a PhD.

    However, it’s not clear to me that PubMed would be an ideal resource for an MD who’s mainly interested in the current best consensus on any given disease or conditions. Most of the stuff that gets published in the research literature is the cutting edge stuff. E.g., our clinical trial found that such and such drug appeared to improve disease X in these patients. That’s important stuff, but not the sort of thing I’d expect an MD to base typical treatment decisions on. I assume MDs need something a bit more authoritative, that reviews all the literature and informs them that yes, such and such drug has now been found to work for disease X repeatedly, and should be added to the standard of care. Or, such and such drug shows promise, but it’s still to early to consider it proven.

    Is that more like what uptodate offers? If so, I wonder how Starman et al. would have scored it for quality of information. Also, $500/yr isn’t trivial for an individual, but it’s not so much that an academic or non-profit site couldn’t subscribe to ensure that their health info web site is completely current.

    So I guess I’m still a bit surprised that none of the reviewed sites did better than 61%.

  10. Zoe237 says:

    Is it possible for a recap of the presentation on the site? Sounds like an extremely helpful information. I couldn’t access the article for less than $30. Also, some coverage of Avandia would be nice.

    Just checked the TAM website… Dawkins, Shermer, Savage. Drool.

    However, it’s not clear to me that PubMed would be an ideal resource for an MD who’s mainly interested in the current best consensus on any given disease or conditions.”

    You know, I’m not super familiar with the medical field, but I expect doctors to keep abreast of the latest research in their field. On a “primary source” level too. At least, my ob and orthopedist could quote recent research to me. Is that unusual? My pedi couldn’t, and I switched to one who could.

    I would love to find an all inclusive source of information in between pubmed and the crap that is webmd. I like emedicine but it’s too much for a quick lookup for a layperson. What about cochrane review?

  11. Charon says:

    It’s not clear to me that the web is the best place for the layperson to turn for general medical information. I own the Johns Hopkins Family Medical Book, and it seems pretty ideal for such use. As a book, its content can be very thoroughly reviewed. You can look up signs and symptoms, get some tentative idea of what’s wrong with you/your child/whomever, and get advice on whether or not to go see a doctor about it. I doubt such information really changes all that quickly.

    If people want in-depth information about some new chemotherapy treatment or something, then such books obviously aren’t the place to go. But presumably at that point there are experts to turn to for reading recommendations, like one’s oncologist. Those with literature research chops can try going off on their own. If medicine is like other scientific fields, the progression I’ve often found when looking outside my own area of expertise is general textbook -> specialized textbook or monograph -> recent journal articles, as one hones in on what one is looking for.

    Not that there isn’t some role for the web here, but maybe it’s not always the best way to go. That said, I thoroughly enjoy SBM on a regular basis :)

  12. Charon says:

    And yes, I know my comment is slightly off topic – this post/thread is a discussion of where to find good info on the web, and what quality it is – but I think it’s good to remind people that sometimes the web is not the place to go.

  13. RachelW says:

    Medical librarians are also a great source of information on consumer health resources that are relatively good/up to date/transparent/reliable. One place I often tell people to start is MedlinePlus, which is run by the National Library of Medicine and compiles links to information on a wide variety of health topics, and has abundant Spanish-language and other non-English materials as well. Most topics have a general overview, and then links to more detailed info from professional medical organizations, government sources (such as CDC consumer-oriented pages), and so on. As an example, here’s the main page on ankle injuries –

    There are a few other sites I really like for consumer health info, and some strategies medlibs generally talk about for evaluating online content, if anybody is interested in talking about that.

  14. aeauooo says:

    On the subject of subscription-based services; in addition to journal literature searches, MDConsult includes the full texts of roughly 30 medical textbooks, patient education materials, clinical guidelines, and drug information.

    I’ve been told that I’m old-fashioned because I prefer Ovid to PubMed.

  15. RachelW says:

    aeauooo, your Ovid comment made me laugh. I know that we don’t teach the Ovid version of MEDLINE to our med students and residents b/c we know they will be able to access the PubMed version anywhere they go, but may not have a subscription to Ovid.

  16. Joe says:

    @RachelW on 15 Jul 2010 at 12:54 pm The NLM and NIH subscribe to unreliable sources of information about so-called alternative medicine and “dietary supplements.”

  17. RachelW says:

    Joe, do you have a couple of examples you’re concerned about? I’m not with NLM/NIH, but would like to know what you’ve seen that was problematic, and also encourage you to contact them about it.

  18. Newcoaster says:

    As a part-time family doc, part time ER doc I have need for different kinds of information depending on what setting I’m working in. In the ER I need short and to the point information..the pearls. In the office, I can afford a bit more time to get more background on the new patient with the disease I’ve never heard of.

    I like e-medicine because the articles are to the point, and I can usually skim them quickly for the key points, but they also have references for more detailed reading. I’m a recent convert to UpToDate, especially when I’m wearing my family doctor hat. I kept seeing all the newly minted docs using it, and finally bit the bullet and paid my $500 subscription. It is a big chunk, but at least it is “write-offable”.

    I get a lot of the free (aka “junk”) journals that are heavy on advertising and have short summary articles or viewer questions, but rarely read them. I wouldn’t use that information as a definitive source without checking elsewhere.
    I rarely use PubMed/Medline or Cochrane except when I am debating Alties online, or trying to show patients that there is no good evidence for whatever potion the Naturopath has put them on this month.

    My beautiful 2 volume set of Harrison’s still sits on the bookshelf in my home office, gathering dust pretty much since I graduated med school 15 years ago. It and most of the other med school era textbooks are there only to impress people, as I can’t remember the last time I cracked one of them.

Comments are closed.