Articles

CureCrowd – Crowdsourcing Science

curecrowd-HAA new website came to my attention that promises to “Discover what truly works.” The idea is to essentially crowdsource anecdotal reports about what treatments work for specific conditions.

This is an interesting idea, that can harness the power of information flow over the internet to do what is essentially an observational, uncontrolled, and unscientific study about treatment effects. At its best, this type of information would consist of what we call a pragmatic study – an open-label study of the real-world application of specific treatments.

Pragmatic studies have their place – they add information about the practical applicability of various treatments. Pragmatic studies, however, are not efficacy trials – they cannot and should not be used to make efficacy claims. It has recently come into vogue for proponents of various CAM treatments to rely on pragmatic studies to make efficacy claims, when actual efficacy trials have failed to show effectiveness.

Pragmatic studies, in other words, should be conducted only with proven therapies that have already demonstrated efficacy.

The information that would be generated by CureCrowd, however, is more problematic than a well-conducted pragmatic study. A good pragmatic study should, at least, be systematic – looking at consecutive patients in a practice, for example. Online surveys, however, are self-selective. This introduces selection bias which may overwhelm the results. Therefore the information may tell us about what people feel motivated to report, not actual results.

It does, however, give the illusion of useful information. Take a look at the graph at the top of this post – results so far for headache (the site is just getting started and so the information is still scant). I treat headaches so I can put this information into context. You can see that caffeine is reported as reasonably effective. However, it’s possible that for most people who believe caffeine treats their headaches, they are just treating headaches caused by caffeine withdrawal. For most people with recurrent headaches it is best to completely avoid caffeine, but this chart may make them think caffeine is an effective treatment, leading to more consumption.

I should also point out that “headache” is a problematic category. There are many headache types, such as migraine vs. tension headache. This information is not useful without knowing what kind of headache the different users were treating. This brings up the broader problem with using an online survey for medical information – who verifies the diagnosis being treated?

Conclusion

I do think that using the internet to generate large amounts of possibly-useful medical data is an interesting idea which could have some potential. But there are many pitfalls. Just gathering large amounts of data will not necessarily tell us anything useful, and in fact may be misleading. It can give the false sense of useful information even when results might be grossly misleading.

Treatment effects are particularly problematic, as they are susceptible to placebo effects and a host of perception and reporting biases.

Scientists use uncontrolled observational studies all the time, but they have to be put into their proper context. They are useful for generating hypotheses, not so much for definitively testing hypotheses. They are also useful follow-up data to see how proven-effective treatments work in real-world settings.

An online survey might generate some interesting data, but I don’t think it can be relied upon to “figure out what really works.” The fatal flaw in this approach is the self-selective nature of the information.

My major concern about this type of project is that it is designed to give information back to users, with the promise that this information will be useful. Rather, it would be more legitimate to use such a website, at least initially, as an experiment itself. First let’s validate this type of information. The online surveys could be done and the information generated could be compared to controlled efficacy data. If the results track well with controlled data (where such data is available) that would at least tell us something.

A great deal more thought needs to go into this type of crowdsourcing, such as how to capture more accurate and detailed diagnostic information. “Headache” is an ultimately useless category. Users should at least be given the option of filling out a standard headache questionnaire, and that information can be analyzed separately.

Using online methods for capturing health information is a potentially useful new tool for medical scientists, but a great deal of thought and analysis needs to be done to develop such tools before they will be actually useful. Treatment information is particularly difficult. Giving such raw data back to users, at this stage, may be more harmful than beneficial.

Posted in: Clinical Trials, Computers & Internet

Leave a Comment (21) ↓

21 thoughts on “CureCrowd – Crowdsourcing Science

  1. Windriven says:

    Appeal to popularity masquerading as crowd based science.

  2. Michael says:

    Trans-dental electromicide worked wonders for my Bonus Eruptus.

  3. Andysnat says:

    Some years ago, I had cause to have a course of treatment with Fludarabine and Cyclophosphamide, which did me the world of good. My lymphocyte counts on the blood tests went from 170k back to approximately 4k during the treatment, a marvelous thing to be part of as you will appreciate.

    Coincidentally I began to partake of three small glasses of (cheap) whisky each evening at the same time as my treatment began.

    In those murky corners of the internet where people with my condition gather and suggest stupid treatments to each other, I often tell them of my whisky cure, while neglecting to mention the chemo, and ask them to learn the difference between anecdote and evidence.

  4. Frederick says:

    Excellent analysis, But, what about crowd sourcing REAL research : clinical trials, Physics experiments, space probes etc. That would be a better idea
    but I guess this idea could be less popular because the backers might not have the info right away ( since studies that got money for CS will still need to be published in peer reviewed journals of course) .
    Also we can anticipate that sCAMer will use that web site to build “proof” of what they are doing. And since cam is more like a religion, people blindly believes in it, they will have no problems finding backers that will throw money into it, and fill up the surveys with their biased minds. and the cam seller with use it, and claim that they have done science. So I’m pretty skeptical of how it will be used in the real world.

    Not than I’m against crowd sourcing, for tech and others project ( as a early backer of Star Citizen, I’m totally for Crowd Sourcing : ) ). But for science it need more rigorous rules.

  5. David K says:

    I can’t take this seriously at all. Maybe it will turn out to be a psychological experiment. This will provide some good laughs for the rest of my day. Thank you for bringing it to my attention.

  6. CatC in Michigan says:

    Having been harmed by both the fed EHR mandate AND the insurance companies controlling my doctors, can crowd-sourcing be any worse ???

    Detail: When I was handed an exit bill and 1-page ‘patient summary’ of newly entered EHR data (after 25 yrs annual appts at the same complex), half the items were totally wrong — conditions that never happened and weren’t mine! Entered by their disgruntled Admin staff, my records are now garbage, then I was told I’d have to pay for an appt to fix their ‘errors’. Why didn’t the practice contract a reputable data entry contractor with the normal 1/1000 error rate to enter/verify their 25,000+ patients ? I hope no one is injured or killed by them!
    Meanwhile, my long-term cheap maintenance drug was dropped by BCBS after 10 years, so a replacement scrip’s side effects (very high BP etc) landed me in the ER for three days… How cost-effective was that ?!

    1. Windriven says:

      “Entered by their disgruntled Admin staff, my records are now garbage, then I was told I’d have to pay for an appt to fix their ‘errors’.”

      I suspect that a respectful but firm letter to the practice administrator or a conversation with your physician will resolve the ‘pay for an appointment to fix their errors’ problem. If not, I’d suggest an appointment with your local television news department’s consumer advocate. Many of them have one these days because it makes for good human interest stories and lets everyone know that 7 News is on their side.

      Your point is well taken that errors in the EHR are potentially very dangerous.

    2. MTDocw says:

      Your points are well taken. You apparently have some knowledge of what should be in your medical file. Good for you. Just think of the poor soul that trusts this system! As for me, I have a paper file of all my pertinent medical information, and it is in my personal possession. I am just now requesting the OP report and Path report of my recent colonoscopy (three polyps) as I want to see details that are not in the physicians recommendation letter to me. I suppose that is the reason doctors are often considered a pain in the rear as patients. As a physician, I dictated detailed office notes, op reports, and letters for 50 years. Now I am supposed to spend my time pecking out my thoughts on a computer, to go into some grand data base which will be available for all the world to see. You can be sure that I will be very careful and brief, and especially limited in expressing any opinions. Just a reminder: Physicians originally wrote notes to remind themselves of details they needed to provide followup care for their patients. I know we have moved far beyond that, but just a historical reminder. As an aside, when office notes were private, some of them made very interesting reading. Well, I’ve probably already said too much.

      1. Windriven says:

        @MTDoc

        ” I am just now requesting the OP report and Path report of my recent colonoscopy (three polyps) as I want to see details that are not in the physicians recommendation letter to me. I suppose that is the reason doctors are often considered a pain in the rear as patients.”

        If you’re a pain in the rear, it seems some proctologist returned the favor. ;-)

      2. Andrey Pavlov says:

        Physicians originally wrote notes to remind themselves of details they needed to provide followup care for their patients. I know we have moved far beyond that, but just a historical reminder

        The history is actually rather interesting. This video of Lawrence Weed talking to his colleagues about the importance of notes and how to do them properly is really a cool history lesson. You might enjoy it.

  7. steney01 says:

    On the scale of perniciousness, this ranks somewhat low for me. Now if I thought the medical community could roll out a crowdsourcing symptoms/treatment tracker and have it be adopted by serious people then I could see a lot of potential value. I just feel like the moment you tried to tell people to provide medical info in a serious way for the benefit of science you’d just get a lot of clamor about the evils of the pharmaceutical industry and concerns about insurance companies stealing the data to up your policy price and not a lot of participation. I think googlehealth was a program like this that recently went under do to lack of participation and valuable data.
    Unfortunately I think this type of thing will remain something akin to the casual sharing of advice among neighbors or family members on how to best get a newborn to stop crying. Only your neighborhood now consists of everyone.
    One thing along these lines that will be more widespread and will be interesting to follow is the mass public release of patient reviews of healthcare provider services.

  8. EBMOD says:

    So do they have any controls in place for those with motivated agendas trying to plant false data? I see this just being a place for the ‘Thinking Mom’s Revolution’ types to spam their groupthink…

  9. healthnut says:

    There is no perfect way of deciding what treatments work. RTCs can be expensive and can take years. Doctors can rely partially on their experience — but even a doctor with 20+ years of experience still has seen only a limited number of patients.

    I think medical experts may sometimes feel threatened by the idea of patients educating themselves. So they warn them that information on the internet is not very reliable.

    Well ok, it isn’t. But what is reliable?

    I do not think the crowdsourcing medical site should be limited to “proven” treatments. What is really proven anyway? And if they are proven, your doctor would probably know about them.

    I don’t worry too much about placebo effects — yes they can happen, but I think they are usually weak. If you are very sick, whatever help you might get from a placebo won’t be anything to rave about.

    And if a treatment really is harmful, it is unlikely that large numbers of people will recommend it.

    And hopefully it will include non-mainstream treatments, because they are the least likely to be comprehensively studied with RCTs.

    I think this website could help doctors as well as patients. I recently saw a doctor for something that is all over the internet, but he knew nothing about it, probably because most doctors have no faith in patient report websites. But I think they should.

    1. WilliamLawrenceUtridge says:

      There is no perfect way of deciding what treatments work. RTCs can be expensive and can take years. Doctors can rely partially on their experience — but even a doctor with 20+ years of experience still has seen only a limited number of patients.

      Meanwhile a patient, who generally has seen only one patient, themselves, is more reliable? Because that’s what this system is – a pile of anecdotes and confirmation bias. At best, possibly hypothesis-generating; at worst, another way for quacks, SCAM supporters and the desperate to drum up questionable support for their preferred interventions.

      I think medical experts may sometimes feel threatened by the idea of patients educating themselves. So they warn them that information on the internet is not very reliable.

      The problem is most patients can’t tell the difference between real information and wrong information. That’s how patients end up denying AIDS is caused by HIV infection, or convinced that vaccines cause autism. Experts aren’t “threatened”, they are experts who see patients deceived by hucksters, shills and quacks because the patients don’t understand enough about science and the biology of their bodies, healthy tissues and diseased tissues to know a credible argument from a specious, money- or fame-grubbing one.

      It takes often decades to become an expert because biology is not simple. Patients go to experts because they don’t have decades. To pretend you can become one given a few hours, or even months, on the internet, is wrong.

      And that isn’t even getting into scientific methodology, cognitive biases and how the mind can deceive.

      Well ok, it isn’t. But what is reliable?

      Double-blind, randomized, controlled trials with adequate numbers and low drop-out rates. Expensive and slow, but also the best offer of certainty. This is not.

      What is really proven anyway? And if they are proven, your doctor would probably know about them

      The problem is patients will use this cite for things like the Liberation Procedure for MS, which patients swear works but whose efficacy mysteriously vanishes when blinding and control groups are used.

      I don’t worry too much about placebo effects — yes they can happen, but I think they are usually weak. If you are very sick, whatever help you might get from a placebo won’t be anything to rave about.

      You don’t worry about it too much, but that may be because you don’t appreciate it enough. Placebo can make pain disappear. Soldiers can go from writhing in agony from a gut-shot to serenly calm because of saline. You can walk on a severed stump, have teeth extracted and make agonizing phantom limbs vanish with placebo effects. They can be immensely strong, and there is a tremendous amount of evidence for this. Bloodletting and acupuncture persisted for milennia because of placebo effects, homeopathy for centuries.

      And if a treatment really is harmful, it is unlikely that large numbers of people will recommend it.

      Again I will point to bloodletting, as well as, for a more contemporary example, how about laetrile? A handy way to get cyanide poisoning and nothing else. Purges, laxatives to cure autointoxication, the list goes on. The human brain is not good at arriving at correct information that is not immediately apparent.

      And hopefully it will include non-mainstream treatments, because they are the least likely to be comprehensively studied with RCTs.

      That’s a bug, not a feature. Non-mainstream treatments are often not studied because they are incredibly improbable. Using this to drum up spurious “evidence” is a bad thing, not a good thing. Non-mainstream, particularly those not being investigated, are often not investigated because for them to work our understanding of the disease in question would have to be wildly, amazingly wrong. Cancer would have to be caused by acidic blood rather than altered DNA, drug concentrations would need to be irrelevant, and so on. Non-mainstream ideas are non-mainstream not out of prejudice, but because they simply do not align with the massive amount of information we know already about health and disease, and proponents refuse to acknowledge this body of work. Often they do not even appear to be aware of it. Patients even less so. This survey would in general add to that problem, not help, and we’ll end up with more medical conditions like Chronic Lyme Disease, that are forced into medical care by political action rather than good-quality research.

  10. chris hickie says:

    Crowdsourcing why I can’t get my wi-fi to work on smartphone I just rooted seems reasonable. Get a bunch of tech-minded people together and someone will figure out where to root around in the OS settings to get the wi-fi back on–or at worst I brick my phone.

    Crowdsourcing health-related issues to rely on anectdotal reports does not appear to be a good idea, especially if the concept gets a false legitimacy. My concern would be people would take what is on these crowdsource sites as fact and, without consulting their physician, do something that could cause harm to their self.

    1. WilliamLawrenceUtridge says:

      Particularly given your phone working or not has multiple issues that make human judgement unnecessary. Your phone either connects to wi-fi or does not. The experience is not subjective. The results are immediate. There is a well-defined mechanism that is human-created, not evoluionarily cobbled-together.

      Illness is none of these things.

      Uncontrolled crowdsourcing is a terrible idea for anything but hypothesis generation.

  11. EBMOD says:

    “I think medical experts may sometimes feel threatened by the idea of patients educating themselves. So they warn them that information on the internet is not very reliable.”

    With my patients, I echo what William put so well: it isn’t us being threatened, it is us being enraged by the ignorance of arrogance when self-proclaimed experts on the internet lead people astray.

    Whenever possible, I print out studies for my patients, and try to explain to them the reasoning behind a given treatment. I also tell that that science is supposed to be empowering, it is the way for lay people to know what is real and what isn’t. I ask them to embrace it. As WIlliam stated, they just need to know what true science looks like.

  12. Catherine says:

    In a world where everyone told the truth, the whole truth and nothing but the truth, this could be incredibly useful.
    I’ve been a patient for headaches and also for infertility. In the offline world and in other places online, I have occasion to interact with someone who has one of the same afflictions or someone who knows someone who does and they will relate what worked for them. Or what they think worked, when the problem got better in spite of what they did. From these sources, I’m getting anecdotal data that has an even greater selection bias than crowd-sourced data would have; this includes the referral from the fertility clinic to their favorite acupuncturist. How acupuncture for me was going to prompt my husband’s body to start making sperm is a mystery even they couldn’t begin to explain. No doubt the lure of insurance coverage and cash from desperate patients does inspire the worst in that profession. My point is there are many long and winding roads to dead ends and promises that this next thing will work, and as a patient, you believe it after you are already somewhat invested or it worked for a friend of a friend. Having a dashboard with the 5 or 10 or 15 treatment options upfront and capturing data from a large enough sample size could eventually show something close to what the best research also shows — that some results people fervently believe in are statistical noise.

    The big pitfall that I see, isn’t that a headache sufferer will keep drinking coffee. I am, and I do. The pitfalls are that the data can be manipulated by scammers and by well intentioned folks reporting short term success that is ultimately incorrect. So far as I can see, there is no way to prevent an avalanche of people reporting that juicing from GMO free veggies cured their ______________ right up until their hair fell out or they died.

    1. Catherine says:

      Those bald vegans don’t come back online with a mea culpa. Neither do dead people.

  13. Jazi Zilber says:

    If these reports will be collected automatically (via iPhone etc.) And decided to be collected before treatment it will be more usefuk

Comments are closed.