Researching SBM Online

The internet is a fabulous resource of information. It is one of those technological innovations for which you soon can no longer imagine how you lived without it. I certainly cannot imagine a project like science-based medicine prior to the web.

The web, however, is also a tremendous source of misinformation, opinion, and ideology. Also the volume of information, good and bad, can be overwhelming. We therefore are frequently asked the meta-question of how we conduct our research into specific topics, or how can the average layperson do their own research online.

Efficiently and effectively researching a complex topic is complex. It is a skill that needs to be developed, and it is especially difficult without having detailed knowledge of the specific topic ahead of time. Therefore there is no simple answer to this question, but I can offer some tips.

There are two main resources I use when searching a topic, Google and PubMed. They each have their strengths and weaknesses. For the average user, Google (or whichever general search engine you prefer) is likely going to be your first stop.

Using Google

When I am researching an unfamiliar topic the first thing to pay attention to is the search terms. You may not be using the proper or most precise terms in your search because you are unfamiliar with the jargon. You may need to keep searching with variations of descriptions of the topic in which you are interested. If you know the name of someone or an institution attached to that topic, include that in the search.

When you hit an article that refers to or is related to the topic, pay close attention to the specific terms that are used. They will become your new search terms as you hone in your search. Sometimes it take me several rounds of doing this before I get to the precise terms that get me to the content I want.

The strength of Google is its search capability. The weakness is that the results are going to be contaminated with many low-quality sites from an SBM perspective – meaning sites that are commercial or ideological. Once you find your optimal search terms, adding “skeptic” or “skeptical” to the search may bring you to the critical analysis for which you are looking.

Using PubMed

PubMed is a publicly available database of peer-reviewed journals in the medical field, including basic science and psychology. Overall this is a better resource than Google, but you need a higher level of familiarity with the topic. Often I use Google just to find the proper search terms to use in a PubMed search.

The strengths of PubMed include the fact that you are taken only to peer-reviewed primary scientific sources. The search engine, however, is not as robust or user-friendly as Google, so you really do need to know the technical search terms. You can also, however, search by the name of the publishing author or the journal, if you have that information.

An imprecise search can result in tens of thousands of results. You may have to wade through dozens of articles to find one that is relevant to your topic, but once you hit one relevant article that will likely lead you to the articles you want through better searches. Also, with PubMed, when you click on an article the site will also list other related articles. Look through those, and when you click on them you will get more related articles.

PubMed displays the abstracts of articles only. Abstracts are useful summaries, and may be adequate for your purposes  – if all you want is to learn the bottom line. But for any detailed analysis you need to read the full article, as abstracts are often misleading. PubMed offers links to full articles when available, which is increasingly the case, but often the full article is not available online or is behind a paywall (an expensive paywall).

A well-written article will often contain an introduction which provides the authors’ summary of the topic, including an overview of current research. This is often an excellent way to understand the current thinking of researchers about that topic.

The most useful articles on PubMed are often going to be systematic reviews. If you want to go straight to these articles, then just include the word “review” in your search. Reading the three or four most recent systematic reviews of a specific topic or question is the quickest way to get a useful bottom-line summary of the status of the evidence.

Evaluating Resources

Not all resources are equally valid or useful. When it comes to published primary research, the reputation of the journal and the authors are the most important criteria.  That is why I will search on the author names (in both Google and PubMed) to find out what other research they have done, or if they are involved in any ideological advocacy.

Luckily journal names often give away their focus and impact. Journals with names like “Nature” and “Science” have a long history and solid reputation. Journals with names like, “The Filipino Journal of Cancer Homeopathy,”  likely don’t have the same reputation. But if in doubt, search for other articles in the same journal to see what they are all about.

When evaluating sites on the web, there are a few useful rules of thumb. Group sites are more reliable than sites dedicated to the work of a single individual. University sites tend to be of higher quality than commercial sites. Be suspicious of sites that are trying to sell you something, or that have a clear ideological ax to grind.

Otherwise, the general principles of skeptical analysis apply. Be wary of sites that are promoting conspiracy or fear mongering, or that are making claims that seem too good to be true.

In addition to SBM, another deep resource of medical information on the web is QuackWatch. On the homepage there is a section for “Nonrecommended Sources of Health Advice.” That is an excellent list of sources to avoid, unless of course you are looking for claims to debunk. There is also a list of recommended links.

Putting it all together

Perhaps the most difficult question when researching a specific topic is this – when are you done? When have you found all the relevant information? You may feel as if you have found many valuable resources, but perhaps you are missing that one article that changes everything. Again, there is no simple answer to this. The more time you invest in research, the more confident you can be that you have found all relevant information.

The only way to be sure is to do a systematic search, but that is frankly not feasible for the average person. Even for experts, like the authors of SBM, we rely upon other researchers to do systematic reviews, which can take hundreds of hours.

But, following the tips above, if you are reasonably diligent, you can wrap your head around a narrow topic or specific question with a few hours of online research.

It is always useful, however, to simply ask another person for their opinion, especially if they have more expertise in that area. I will often ask other experts – this is what I have found, am I missing anything?

One of the advantages of blogging is that it serves as a means of crowdsourcing. If, after doing my research, I did miss something, it is likely to pop up in the comments. The only way to be really sure you thoroughly covered a topic is to discuss it with others who have knowledge in the area. You have to engage with the community.  That is part of the reason why scientists go to conferences and meetings.

If you are not a blogger, then simply engaging in the comments of a blog like SBM can serve the same purpose. You can also ask questions on forums and message boards. Of course, such online communities are likely to have their own culture and biases. You can easily get sucked into an echo chamber with an extreme point of view. Keeping your sources diverse is important here as well.

In truth the process of researching a topic is never truly done. Even for topics that I feel I have mastered, I am frequently encountering new viewpoints or nuances. Like science itself, it is an endless self-corrective process that requires you to be open to revision.

Posted in: Critical Thinking, Science and Medicine

Leave a Comment (16) ↓

16 thoughts on “Researching SBM Online

  1. Ash Simmonds says:

    What do you think of Google Scholar?

    I don’t do anywhere near the amount of science searches as you, but in general I’ve found Google to return more relevant results with less keyword build-up.

    1. Pharmacist-in-Exile says:

      GoogleScholar is a fantastic second opinion, or simply for delving deeper into hits from PubMed – but it has its drawbacks… It is not maintained or curated by initiated scholars and thus can retrieve a whole lot of bad science papers (especially from predatory open access publishers), so you need to have your own developed sense of critical thinking and a slightly better understanding of the topic you are researching than if you use curated databases such as PubMed.
      With the state-of-affairs between protagonists and antagonists on the different scepticism-shores usually ending up in a “source war” GoogleScholar is a great resource to actually see where the “opposite” side is coming from (and usually how bad their science is). It is also a good place to actually get hold of a large number of papers if you’re not connected to a university library (or the like) without too much search efforts!

  2. Reade Bricker says:

    I think trust is the core of the issue here. Trusted journals, trusted authors, trusted studies and trusted reviews are key when trying to learn about a new topic or question.

    Ex: I trust Dr Novella, because I have read his work and listened to his podcast and believe that he would not deliberately mislead his audience, but I’m not sure I could identify those experts that he trusts and who might be able to supply more trusted information.

    I think a way to see a “chain of trust” on certain topics would help.

    Ex: I trust Dr Novella on topics of neuroscience, so who does he trust and who also trusts those resources etc… etc…

  3. WilliamLawrenceUtridge says:

    IME, Google scholar has the advantage of including a broader range of sources (pubmed is mostly just medical/biology), includes books, but has the disadvantage of including a lot of website results and lower-quality, non-peer-reviewed but ostensibly “scholarly” results and sources as well. I’m less and less impressed with google scholar these days, quackery and crankery seem to be infiltrating.

    It does have the advantage of often linking to (probably illegal) copies of journal articles. I will often start on pubmed, then switch to google scholar to see if I can get a full-text PDF or HTML. Which I never read or download, that would be illegal. I just like knowing it’s there.

    The downside to pubmed’s “author” search is that it is overly broad – it just looks up last name and first initial(s). If that name is “John Smith” or “Jill Chan”, you get everybody with the name “Smith J” or “Chan J”. One of its more frustrating aspects.

    On the other hand, it has a series of extremely handy filters to the left of the search results, with some very good granularity – reviews, systematic reviews, meta analysis, publication date range, species, full text or abstract available and more.

    Personally, I like to include a google book search if I’m really trying to dig into a topic. It also has some nice filters, is mostly restricted to books (with the occasional periodical), and often will include a limited or even full preview with “search inside” capability. Extremely handy! The downside is again, no filter for quality. It also seems to pool your page access with everyone else in your IP – so if each IP only gets 10% of the total number of pages per day, and your IP services 1,000 people, you’re lucky to get what you want. Logging into your google (i.e. gmail) account seems to help considerably since you don’t seem to share it anymore.

    When I’m looking for health topics, particularly introductory and broad overviews, I will always revert to a couple default pages:

    General nuttery
    * the Skeptic’s Dictionary (good for nonmedical topics too)


    Basic advice
    * eMedicine
    * the Mayo Clinic (good for mainstream, getting worse for CAM)
    * the NIH
    * the CDC, FDA and as a Canadian, Health Canada and the Public Health Agency of Canada

    The downside to all of these sources is that they are repetitive – it’s basic information and advice, starting points about as good as a well-written wikipedia article. Speaking of wikipedia, it’s a reasonable starting point but really is best just to mine for sources. Though I do heartily recommend their content guideline for identifying reliable medical sources.

    Medical quackery and nuttery
    * SBM and Quackwatch (of course)
    * Snopes for the kind of absurd claims that even SBM and Quackwatch refuse to dignify with attention.
    * Paul Ingraham’s save yourself website is very good for biomechanical stuff, a topic area I have a hard time finding good information on without having to read a lot of primary literature.

  4. Harriet Hall says:

    2 thoughts:
    I have pretty much given up using the search terms “skeptic” and “skeptical” because they most often turn out to be testimonials. The Internet has been flooded with comments like “I was a skeptic, but I tried it and it really worked for me.”

    On PubMed, there is a center column of “PubMed tools,” and clicking on the “Clinical queries” option is handy for limiting searches.

    1. Stephen H says:

      One hopes the term “sceptic” is also included in searches – although rare, this version of the spelling does exist “out there”.

  5. WilliamLawrenceUtridge says:

    I have pretty much given up using the search terms “skeptic” and “skeptical” because they most often turn out to be testimonials. The Internet has been flooded with comments like “I was a skeptic, but I tried it and it really worked for me.”

    This actually appears to be a specific strategy adopted by many of the crap SCAM claims out there. I’ve run into this problem with things as diverse as acai juice, weight loss, supplements, magical memory magic, piano learning techniques, second language programs and others. They’re all basically clones of the same setup – fake blogs, fake websites, fake news stories, all with single-page sites and a very frequently a similar word salad. I suspect there is some quack marketing company out there making a bundle off of the webpage equivalent of spam.

  6. Jeremy Praay says:

    Adding “skeptic” to searches may be good for researching homeopathy, but when researching other topics, it just tends to muddy the waters even further.

  7. Carl says:

    Wikipedia is good for finding more search terms to use in PubMed because it will usually say “(also called ___, _____, or ___)” in the intro, while the PubMed abstracts may only refer to something by one of several names.

    You can of course use the “site:___” google search if a website has a horrid search system (e.g., “bunko root”), but you can also just do it as to search only university websites or, etc.

  8. Jimmy says:

    The Internet is a horrible place if you have an autoimmune disorder and are trying to find more in-depth information about it–especially if it involves the joints and/or bones.

    The credible websites that WilliamLawrenceUtridge listed are great for basic information like he mentioned. Beyond those sites and the information they provide you start to get into quack territory real fast.

    “Reverse and cure your condition by following a simple diet.”

    “Chiropractic can correct the joint problems of disorder.”

    It’s bad enough having the disease; seeing all that garbage on the Internet just adds insult to injury. It makes me so angry….

  9. angorarabbit says:

    If I could add to excellent comments, it is to read a college-level textbook on the topic. These give good background and define core information that can really help in evaluating a claim. Amazon is a great textbook resource. As MOOCs grow these will become available also.

    When I need to research a new medical topic, I start with PubMed and grab a few review articles. In the old (pre-internet) days, one good review article would open an entire field, because the other gem you get are References. Review articles are less likely to trim the reference list than are primary research papers. I second Dr. Novella’s recommendation to read the Introduction if available – it’s a great way to orient yourself to the questions in the field and what that particular paper seeks to answer. Wiki can be helpful if I have zero background and need a quick start, especially helpful if I’m unsure of the search term.

  10. Chris says:

    Every so often it is interesting to put a search term into PubMed and go the last page to see the oldest papers. The history of medicine folks seem to be busy adding to that, because it used to be just titles requiring you to go the medical school library physical stacks in the basement, but now there are fully scanned papers.

    I have had an issue lately with certain folks who keep claiming that measles used to be just a “mild childhood disease” and do not understand why it should be prevented. I did counter that it had to be serious enough to be notifiable by 1912 to be included in US Census data.

    But recently I found this gem from 1914: A STATISTICAL STUDY OF MEASLES (1914). It has some very scary statistics, and the last paragraph has some eerily familiar comments about a doctor/author spouting off misinformation:

    This misinformation is of a kind with that put forth in a popular way by a widely read physician-author, who, in a book entitled “Preventable Diseases,” makes the amazing assertion that “Fortunately neither scarlet fever nor measles usually becomes acutely infectious until the rash appears.” Surely when doctors fail to realize the significance of the well-known facts relating to measles we cannot hope to accomplish the best preventive results without a vigorous and continuous campaign of education as a necessary accompaniment of any and all other measures.

    1. Carl says:

      Of course, for researchers with a preference for the classics, there is always the Royal Society archives:

      We have seen, for some considerable time past, above 100 persons per week in this city and suburbs, taking one week with another, to be carry’d off by this disease; a consideration certainly that ought to dispose us to enter into any measures, by which we may reasonably hope to put some stop to the progress of so cruel a distemper.
      To this purpose, Sir, the method of inoculation, which has lately been introduced among us, is strongly recommended on the one hand, and has been opposed with a great deal of warmth and zeal on the other.
      – James Jurin, 1722

  11. Eugenie Mielczarek says:

    A few additional personal notes supporting Dr. Novella’s valuable post. Forty years ago when I and other solid state physicists were invited by NSF and NIH funding to bring advanced experimental techniques to study some intractable problems in medicine and biology I found myself on another literature planet. We had expected Science, Nature and PUB MED would be the gold standard of interdisciplinary literature. Unfortunately they are not. I learned 1. Read all references listed by the author especially those which he/she uses as support for their own results. Apparently editorial boards of peer reviewed journals skip this check. In one case a key paper listed as a journal was a privately printer flyer masquerading as a journal. This ‘flyer’ was a key reference in an article which claimed that trained practitioner could decrease the growth of cancerous bone cells by waving their hands over them in a L shaped room. This result was published in a peer reviewed PUB MED journal with an impressive board of physicians which refused to retract the article. 2. Keep in mind ; the pressure to ‘publish ’ is more onerous in the medical and biological sciences than in the physical sciences. The year I spent at NIH –on leave- at their request to build a specialized apparatus was an eye-opener. I built a highly specialized functioning system which studied the properties of model hemes and genetically diseased blood. Just a few years later the system was shut down and its scientist dismissed—he couldn’t turn out enough papers fast enough.

    1. Rork says:

      People using information to further their own research need all sorts of extra caution, maybe cause many claims of findings are based on just one experiment, and there’s lots of pressure to put one or two nice results in a paper that the author knows are probably not true, or at least not robust. It can take me 20 hours to review a paper that holds big data, but most reviewers spend only a fraction of that.
      I fully agree that people often say “X does Y (ref 1)” when ref 1 does not really say that, and yeah, it’s often their own paper where the actual finding got better with time.

  12. Kerri says:

    If you really need to access the full text of the article and it is behind a paywall, remember that for the major medical journals your local public library probably has a subscription to a hard copy, if not digital access from home with your library card. University libraries will generally let you access and photocopy the journals even if you are not a student. It’s not always convenient if it’s just one article for curiosity, but if you’re planning to spend the afternoon researching a topic in depth heading out to the library where you can sit down with a medical dictionary and grab the full articles you need can be more efficient than running into paywalls on the internet all day. It’s worth looking into your local options.

Comments are closed.