Articles

Science is hard, and best left to professionals (the same may be said for journalism)

It might seem a bit undemocratic, but science, like medicine or dentistry, is a profession. One doesn’t become a scientist by fiat but by education and training. I am not a scientist. I apply science. My colleague Dr. Gorski is a scientist (as well as physician). He understands in a way that I never will the practical process of science—funding, experimental design, statistics. While I can read and understand scientific studies in my field, I cannot design and run them (but I probably could in a limited way with some additional training). Even reading and understanding journal articles is difficult, and actually takes training (which can be terribly boring, but I sometimes teach it anyway).

So when I read a newspaper article about science or medicine, I usually end up disappointed—sometimes with the science, and sometimes with the reporting. A recent newspaper article made me weep for both. Local newspapers serve an important role in covering news in smaller communities, and are often jumping off points for young, talented journalists. Or sometimes, not so much.

The article was in the Darien (CT) Times. The headline reads, in part, “surveys refute national Lyme disease findings.” Epidemiologic studies, such as surveys, are very tricky. They require a firm grounding in statistics, among other things. You must know what kind of question to ask, how many people to ask, how to choose these people, etc, etc, etc. So what institution conducted this groundbreaking survey on Lyme disease?

Actually, they are quoting the famous work of one Kent Haydock, chairman of the Deer Management Committee. But I’m sure he outlined his methods carefully. Or not.

Haydock conducted:

[T]wo surveys — which polled 41 Darien households after a showing of the Lyme Disease film, Under Our Skin, at the Darien Library last month… . In the 41 households that completed the questionnaire, 47 total Lyme disease cases were reported. In 64 percent of those cases, the patient had relapses after an initial Lyme treatment, which required additional treatment for a chronic or long-term conditions.

So, Haydock showed the agitprop chronic Lyme advocacy film Under Our Skin to local families, presumably not selected at random, and then asked them if they had signs of Lyme disease and if it was ruining their lives. Not surprisingly, the answers to both questions were “yes” a remarkably high percentage of the time.

His conclusion: the surveys “show that Lyme not only exists in great numbers, but also in debilitating, chronic and long-term cases.”

This is not epidemiology. This is not science. This is an uninformed opinion dressed up with meaningless numbers. If you get together a group of people who are interested in Lyme disease, show them a propaganda film, and query them about it, the only thing you’ve “measured” is your ability to count people who come to a movie and hold a certain belief. If there were any valid conclusions to be drawn (and with these numbers, there probably aren’t) it’s that many people in this small group think they have Lyme disease—and even that’s over-reaching.

It’s bad enough that the deer commissioner did this. But arguably, it’s much worse that the reporter and editor published it. The only thing this accomplishes is fanning the fears of the readers.

Posted in: Science and Medicine, Science and the Media

Leave a Comment (24) ↓

24 thoughts on “Science is hard, and best left to professionals (the same may be said for journalism)

  1. daedalus2u says:

    I think you sell yourself short. I think you are a good enough scientist and a good enough journalist. You can be good enough at what ever you do precisely because you are a professional.

    To me, the quintessence of being a professional is to know your limits. As Clint Eastwood says, “A man’s got to know his limitations.” If you know your limits, then you won’t try to do things that are beyond them. That makes you good enough to do what ever you can do.

    Knowing one’s limits is what keeps angels from treading where fools are willing to rush in. Unless you are pretty knowledgeable in a field, it can be difficult to differentiate confidence from bravado from arrogance and ignorance from unwillingness to jump to a conclusion based on insufficient information.

  2. Brian Robinson says:

    The problem with this article is that it was reported as a local news story and not as a science story. That’s often the bane of science reporting at the local level. It points to local companies and to local outlets, but doesn’t cover the science, not least because local journalists sent to cover these stories are not science reporters but local reporters who cover everything from council meetings to police reports.

    However, that doesn’t excuse the lax reporting here. One phone call to any number of colleges or universities would have popped up an outside authority who could have commented on the “findings” of this survey, or at least on the methodology, or lack of it.

    Not sure how you get around this, other than better education of journalists about science and the scientific process as a basic part of their education. I got a science degree before I went into the business, and that’s the only thing that’s helped me get around a lot of the intricacies of science. Even then, I get enough wrong.

    Unfortunately, as evidenced by the fact that newspapers are dumping their standalone science sections, science just doesn’t seem to be any priority with the general press today.

  3. Dubito says:

    This was a fantastic post, Peter, thank you.

    I’ve got an Honours degree in Psychology, but haven’t worked in/with science until I needed to revist all my research skills as an informatin officer in a disability service, and what I found horrified me so much that I’m now studying journalism because I think I can report medicine better than it’s being done.

    I think your first paragraph needs to be engraved on many people’s frontal lobes.

  4. pec says:

    You neglected to mention this:

    [To corroborate the library survey, which Haydock conceded may be presumed to be weighted toward long-suffering families, a second survey was conducted a week later at a meeting of the Darien Senior Men’s Association.

    In that survey, 34 of the 71 men that completed the questionnaire reported cases of Lyme disease in their household. Of those, 22 percent reported a relapse after initial treatment.]

  5. Fred Dagg says:

    This is a multi-choice question. Please answer it to your best ability. There is no right answer.

    1). How much of “Medicine” today is purely “Evidence Based”?

    a). 0 to 15%
    b). 16 to 30%
    c). 31 to 45%
    d). 46 to 60%
    e). In excess of 61%

  6. Peter Lipson says:

    pec, translated, h/t HH:

    To corroborate the survey, which may be presumed to be weighted toward certain families, a second survey was conducted a week later at a meeting the Fourth Street Gymboree.

    In that survey, 34 of the 71 children who answered reported the Tooth Fairy visiting their household. Of those, 22 percent reported a return after initial visit.

  7. Harriet Hall says:

    Answering Fred Dagg’s question:
    (f) Much more than ever before
    (g) More every day
    (h) FAR more than any alternative system such as chiropractic.

  8. Fred Dagg says:

    Harriett

    I asked a serious question which deserves more than a glib answer you have managed to provide. If you cannot answer it, and have to resort to snide and sarcastic comments, then it shows a level of your lack of understanding of “Evidence Based” healthcare.

    If you would like to break it down into sub-groups then you are very welcome.

    1) Orthopaedics
    2) Obs and Gynae
    3) Osteopathy
    4) Chiropractic
    5) Psychiatry
    6) Neurology and Neuro-surgery
    7) Opthalmology
    8) General practice
    9) Pharmacology
    10) Genito-urinary
    11) Cosmetic Surgery
    12) Plastic Surgery
    13) Physical-Therapy
    14) Musculo-skeletal medicine
    15) Dermatology
    16) ??????????

    PLease Harriett and others answer my question honestly and with integrity and if you have to rely upon sarcasm, make it good, because it reflects the quality of your answer.

  9. Harriet Hall says:

    Fred Dagg,
    I answered your question honestly the only way I could. I don’t know what the percentage is, and it varies not only by specialty, but by how you define evidence, how you rate the quality of evidence and where you choose to draw the line.
    Your question suggests an underlying agenda. Suppose we answered with one of your choices a-e. What would you say next? What is your point?
    We have never claimed that all of conventional medicine is evidence-based; it clearly isn’t, but we are striving to make it as evidence-based (as science-based) as humanly possible.

  10. Peter Lipson says:

    I’m not sure frankly what you are asking, or what you think you’re asking.

    For example, cardiovascular disease, which kills most americans, has very strong evidence to support most of it’s standard of care. Medical treatment of CHD, CHF, HLP, HTN are all very well-supported by evidence.

    Diabetes, another major health problem, is very well understood as well, and the treatment of DM is well supported by evidence.

    Many of the things we do that don’t have clear evidence are not as important. Also, even when we don’t have a precise answer, we generally follow the evidence. We dont know, for ex, exactly how to treat sinus infections, but we know that most people do not benefit from antibiotics early in the course. While this is not a satisfying answer, it is, in fact evidence-based.

    So rather than asking your overt concern-trolling question that curiously lumps chiropractic with ophthalmology, why don’t you try to point out what areas of medicine concern you viz lacking any support of evidence?

    And while you’re at it, why don’t you propose an alternative? You know, like, freeze like a deer in the headlights until more evidence is available (a “threshold fallacy” if you will), or abandon science altogether for something else.

  11. Fred Dagg says:

    Nice answer, Peter and Harriett

    you have answered as honestly as you can, and I appreciate the effort. I have read that only 20% of healthcare (not “medicine”) is “Evidence Based”. I would appreciate your comments.
    The reason why I asked the question is that throughout this site and all the comments there is an impression one gets from you that “Medicine” provides you with the umbrella of being evidence based , where in fact it is your desire to be evidence based, as is I would say the desire of all health care providers.
    However, the tone of your posts do not reflect that desire and we only need to look at Harrietts “Adverse Effects” post to see that she was not fair in her writing. I believe she formed an opinion and then wrote her article with a conclusion in mind irrespective of what the research said. In other words, she wrote the conclusion first, then based her whole argument around justifying an eronneous conclusion without really looking at the intent and body of the research. I have pointed that out to her in the posts, so we do not need to go there.

    If you would like an area of healthcare that concerns me, then how about the rates of spinal surgery in the U.S.A. in comparison to the United Kingdom?
    How about the use of Asprin in blood thinning. The latest Lancet talked of the cost benefit analysis of this and said that the risks outweighed the benefit.
    How about the use of anti-inflammatories especially the devasting effect Vioxx had in the 5 years on the market. How many people died? What was the cost benefit analysis. These are the types of questions I want answered in “Evidence Based Medicine”, not a pseudo-scientific poorly written and poorly researched article like “Adverse effects………”How about thinking about looking at evidence based care within “Medicine”, rather than evidence base within “Healthcare” or “CAMS”.
    Otherwise you should change the title of this site.

  12. Harriet Hall says:

    “I believe she formed an opinion and then wrote her article with a conclusion in mind irrespective of what the research said.”

    I think I reported the evidence fairly, and you have not shown that I did not. The new study corroborates earlier studies showing the incidence of short-term adverse effects from visits to a chiropractor are around 50%. Do you deny that evidence? Do you have better evidence you can show us?

    Your speculations about my forming an opinion before I looked at the evidence do not deserve an answer.

    It sounds like you want us to criticize only medical doctors and not criticize chiropractors and other alternative practitioners. That was not our purpose in starting this blog. Our purpose was to promote a rigorous scientific approach to medical and health claims of any origin.

    We do too many spinal surgeries in the US, and that is being addressed in the medical literature. The Vioxx story was not as bad as the press made it out to be. In 2005, advisory panels in both the U.S. and Canada encouraged the return of rofecoxib to the market, stating that rofecoxib’s benefits outweighed the risks for some patients. The vote in Canada was 12-1, and the Canadian panel noted that the cardiovascular risks from rofecoxib seemed to be no worse than those from ibuprofen — though the panel recommended that further study was needed for all NSAIDs to fully understand their risk profiles.

    By the way, you mis-read the Lancet study. It addressed primary prevention in low risk individuals. Current medical practice (based on good evidence) is to use aspirin for secondary prevention and to consider it for primary prevention in high risk patients.

    Science is working well to sort out questions like these. We are focusing attention on areas where science is not being so rigorously applied.

  13. Fred Dagg says:

    Hell Harriett,

    the ultimate conclusion of the article is what is the most important part of the research article. You cherry picked to suit your preconceived agenda.

    Prove to me I me that I am wrong. I wonder if the authors of this article would have envisaged that it could be so mis-interpreted, as you did.

  14. pmoran says:

    Oh crap, here it comes again, I think — the tired old plea of chiropractors that they be judged by the WORST that they think they can find within conventional medicine.

    Fred, there is a lot that can be brought up about all these matters, including the many studies that you should easily be able to find showing that about 80% of medical practice of a variety of types is evidence-based to a reasonable standard. Also note that medicine usually acts promptly when the evidence is clear. Has chiropractic ever changed or discarded *anything*?

    So I think it is time chiropractic grew up and looked clearly at itself. It has a very, very, difficult question of its own to answer — “does anything uniquely ‘chiropractic’ have anything to contribute within modern medicine.” (Harriet’s recent question phrased differently.)

  15. daedalus2u says:

    Fred, you are completely wrong, and your statement shows that you don’t understand how to read the scientific literature.

    The most important part of any research article is the data. If the data taking was well done, that data will be good for the rest of time. The conclusions may change as other data in other papers is incorporated. The conclusions may change as the understanding in the field increases. If the data is good (meaning well done and accurate), then all future conclusions in all future papers on the subject will have to account for it.

    If the data is wrong, the paper should be retracted. If the data is wrong the paper should be retracted even if the conclusion is correct.

  16. Psyche78 says:

    I understand the frustration of having research misrepresented or oversimplified by the media. There are times when the mistakes are so blatant, blame can only be placed on the carelessness of the reporter and editor and there is no excuse for that. However, in a many of cases, overreliance on jargon and an inability by the researcher to communicate scientific findings at a level that a non-researcher (when approached by the media, for example) could understand contribute to the poor state of reporting on science. I know that during the late 90s, there was an attempt by journalism programs to address the difficulties on their side by providing classes on science journalism; at the time I received my undergrad, my university was offering a masters degree in science journalism. But when I was in graduate school pursuing my doctorate, in the midst of experimental classes and lectures about presenting findings, not a single lecture/topic was about presentation of findings to the general public or about speaking with reporters. This ability is really one that needs to be taught in grad school, as it would not only be of benefit in helping to communicate findings to the public, but would help in trying in communicating with lawmakers and other gatekeepers of funding when it comes to securing funding.

    @Fred: I understand what you are saying about getting the sense/implication from the compendium of blog articles on this site that “Western medicine = evidence-based” while Eastern medicine does not, and share your frustration about that. But your delivery of that sentiment is less than optimal and will only raise hackles and cause defensiveness. A less argumentative tact might be better received.

  17. Calli Arcale says:

    Fred,

    I understand your frustration. You see that there are failures to be evidence-based in what most would regard as “conventional” medicine, and you wish that more attention was given to these failures. I share that frustration.

    However, you then go on to imply that the bloggers on this site only point their fingers at “alternative” medicine, without ever showing disapproval of “conventional” medicine. If true, this would be a serious allegation (though it would be within their rights to choose to blog about alternative medicine; that is, after all, a target-rich environment with respect to non-evidence-based medicine). Fortunately, it is not true. Perhaps you have not read enough of the posts on this site, but there have certainly been articles pointing out non-evidence-based practices in “conventional” medicine.

    And that’s just it — the bloggers here do not restrict themselves to finger-pointing at their commercial competitors (unlike a lot of alt-med practitioners), but rather take the time to discuss various specific instances of non-evidence-based medicine. You see, they all share the view that there really is no difference between “alternative” and “conventional” medicine. It’s all the same thing — medicine. What matters isn’t where it’s taught or or what cultures produced it or anything silly like that. What matters is whether or not the evidence supports its use.

    Hence, “Science-Based Medicine”. Forget the false dichotomy of “alternative” versus “conventional”. It’s all about what has science behind it and what doesn’t. And there are certainly mainstream practices which lack evidence. These bloggers have already written about many of them, and I have no doubt that they’ll write more.

    Just to pick one such article arbitrarily, Harriet Hall wrote Is IV Sedation Over-Used? She is questioning the widespread practice of using sedation, which carries significant risks, for unpleasant procedures such as bone marrow aspirations, endoscopy, etc. when many patients might be willing and able to tolerate the procedure without sedation if the doctors just took the time to *ask* them before drugging them.

    The mainstream doesn’t get a free pass here just because it’s mainstream. Evidence is demanded of it as well.

  18. Joe says:

    @daedalus2u on 11 Jun 2009 at 5:49 am “Fred, you are completely wrong, and your statement shows that you don’t understand how to read the scientific literature. ”

    You got that right. He thinks a non-expert commission (a lawyer, a school mistress and a chemist) in New Zealand has definitively supported chiropractic.

  19. dt says:

    @Fred Dagg

    “I have read that only 20% of healthcare (not “medicine”) is “Evidence Based”. I would appreciate your comments.”

    I’d like to know your source, and whether you have one for medicine, not just health care.

    I actually think you will find your question has been answered a couple of years ago by Steve Novella.
    http://www.theness.com/neurologicablog/index.php?p=51
    How much modern medicine is evidence based?

    It’s a good read. Perhaps after you have read it, and gone back to the sources, your knowledge on evidence based medicine will be a little rounder and you will be able to pass this on to others. Who knows, maybe you could even correct misinformation you encounter on the internet..

  20. The Blind Watchmaker says:

    So what was this about Lyme Disease again?

  21. dt says:

    Further info on how much medicine is evidence-based:

    http://www.shef.ac.uk/scharr/ir/percent.html

  22. drbuzz0 says:

    Science is best left to the professionals???

    On this I could not disagree more. Science is not a tool for use by the elite in the ivory towers of the great universities and National Laboratories. It is certainly a profession, but it should not be limited to the professionals. Basic scientific methodology is something that every person should have an understanding of and is a tool to decipher both the small and large things in the world.

    If anything, people outside the science professions are not engaged enough in science and lack the understanding of it that they should have. There are certain things that are fundamental and do not require a Ph.D. to understand: For example, that anecdotal evidence is worthless and well controled scientific studies published in open peer reviewed journals with verifiable numbers and statistical analysis are critical.

    If everyone understood this, a large amount of quackery would disappear!

    I am not a great author or writer or poet and I did not go to school for writing.. but I certainly know how to spell my own name and write a sentence! The average person should have a similar understanding of science – at least the basics and an appreciation for why it is so.

    In this case the reporter not only is not a scientist, they don’t know the first damn thing about science. The problem is that they are totally illiterate of the most fundamental aspects of it. ALL journalists should know this stuff. All educated persons should. Is not the goal of a liberal arts system to provide a broad foundation in subjects?

    “We have also arranged things so that almost no one understands science and technology. This is a prescription for disaster. We might get away with it for a while, but sooner or later this combustible mixture of ignorance and power is going to blow up in our faces.”
    ~Carl Sagan

    My own quote: “Science is no more limited to scientists than is beauty limited to artists”

    By the way: regarding Lyme Disease. I know it well. I live in Southern Connecticut not far from the town of Old Lyme and in an area that has about the highest density of it. I’ve had it. My brother has had it. My mom has had it.

    It was the crappiest week and a half of my life having Lyme disease. Damn once those antibiotics kick in I was about ready to sing because I had been in so much pain and tiredness before getting treatment.

    Of course the “chronic” stuff is a load of bologna.

  23. Heraclides says:

    For those who don’t know, the nom-de-plume “Fred Dagg” is the stage name of comedian well-known in New Zealand in the 1970s (see http://en.wikipedia.org/wiki/Fred_Dagg).

    Fred,

    The answers you have received were the most appropriate answers. The relevant question is not the percentage that (some stated measure of) evidence is used, but what is the best system to deriving medical treatments: evidence-based.

  24. Jennifer says:

    You can learn on your own how to crrectly read and interpret scientific studies. Look up Stephan from the whole health source blog. Doctors get no training in how to do this. So not all of them are good at it. The ones who are learned on their own.

Comments are closed.