Articles

Brain-Training Products Useless in Study

The health marketplace has a life of its own, mostly separated from science and evidence. Generally the marketplace gets a hold of an idea and runs with it, before the science is carefully worked out. Since most new ideas in science turn out to be wrong, that means most products will eventually be found to be worthless.

One such idea is that “brain training” can improve overall cognitive function – so of course now there is an industry of products which claim to train your brain. Lumosity (just to pick a random example served up by Google) claims on their website:

Brain Train
SCIENTIFICALLY DESIGNED

* Improve memory and attention
* Shown to improve cognitive function
* Neuroscience based brain training
* Train your brain today

I always enjoy the phrase “scientifically designed” or “scientifically formulated” – they are wonderful marketing phrases that invoke “science” without making any specific claims.

The notion of improving brain function by practicing certain tasks or playing cognitively demanding games is an attractive one. I certainly would like this to be true: get smarter while playing video games – I’ll buy that for a dollar.

It also may or may not be plausible, depending upon how you look at it. What is uncontroversial is that the brain learns, and practice does improve function – no question. But, the evidence also suggests that genetics is a dominant determining factor of overall cognitive function – we all eventually seek our maximum potential. (This does not apply to knowledge or skills, but raw brain power in specific areas.)

Prior studies, both observational and experimental, have shown a correlation with playing certain kinds of games and cognitive ability. For example, one study showed that playing Half Life (a first-person shooter featuring physicist Dr. Gordon Freeman) significantly improved surgical skill performing virtual reality endoscopy.

Another study showed improvement in “executive control functions, such as task switching, working memory, visual short-term memory, and reasoning,” from strategy video game playing. And yet another study showed improvements in visual processing from playing action video games.

But questions remain. For example, how transferable are the skills learned from video games? Do subjects improve only in their ability to play the specific video game and closely related tasks, or does brain function improve in more abstract ways not directly tied to the video game?

Also, how much game playing is necessary? Does benefit come only from playing hours a week, and over months and years, or can more modest “training” be beneficial? And further, is self-selection bias primarily responsible for some of the positive results? In other words, are people who are already superior at certain tasks playing video games because they are good at them and therefore enjoy them more?

With this in mind, Adrian Owen et al. set out to conduct a large study of specific brain training products (not video games optimized for fun) on four standard measures of cognitive function: grammatical reasoning, verbal short-term memory, spatial working memory, and paired associates learning. They randomized 4,678 subjects to three arms – one group was trained on programs designed to enhance reasoning, planning, and problem solving. The second group was trained on memory, attention, visuospatial processing, and mathematical calculations. The third group, the control, was given five obscure knowledge questions and asked to use the internet to find the answers.

Each group was tested at baseline and after six weeks, regardless of how many training sessions they completed, but on average the first “treatment” group completed 28.39 training sessions, compared with 23.86 in experimental group 2 and 18.66 in the control group. At the end of the study there were no statistically significant differences among the three groups – all improved slightly in all four measures.

The mild improvement is almost certainly due to the training effect – this is seen generally in studies of cognitive function or studies that use performance on a task as an outcome – subjects get better just from experience. That is why control groups are always needed.

The study showed that the subjects improved in the tasks on which they were training, but that these improvements did not generalize or translate to the benchmarks of cognitive function. This is a fairly solid study with a clearly negative outcome.

Conclusion

This one study, of course, is not definitive. It is possible that more training is needed before significant benefits are seen. Perhaps video games are more effective because they are more engaging and players will spend more time playing.What this study shows, however, is that products sold as brain training games had no documented benefits after six weeks of use.

Putting this study into the context of the overall research, it does make us more cautious about concluding that there are general cognitive benefits to brain training games or entertainment video games. Benefits are likely to be closely related to the specific tasks involved in training, and not transfer to unrelated tasks.

But there is already enough published evidence showing visual tracking, multitasking, and executive function benefits from action and strategic video games respectively that this study will not be the final word. When there is conflicting research, more study is needed.

This study is most applicable to brain training products, and shows that the marketing claims for these products are not justified. There is very unlikely to be any benefit, or any specific advantage, to “scientifically designed” brain training applications. For now, you are better off just playing a video game.

Long live Gordon Freeman.

Posted in: Neuroscience/Mental Health

Leave a Comment (31) ↓

31 thoughts on “Brain-Training Products Useless in Study

  1. Truckle says:

    Yeh I heard them talk about this on the BBC, and this is a much better summary of the findings, thanks Steve.

    I have to admit I did at least think this one could be plausable, but I thought most of the improvement would come from learning the tests themselves, and the specific questions the tests were asking. This study puts it all down to specific learning rather than cognitive improvement which is interesting.

    Now where’s my crowbar…

  2. windriven says:

    Just as well it doesn’t work – all that training is too hard. Thankfully, science has come to the rescue in the form of an attractively priced pill:

    “Developed by Ray Sahelian, M.D., bestselling author of Mind Boosters: Natural Supplements that Enhance Your Mind, Memory, and Mood

    Mind Power Rx mental enhancement formula is a Dietary Supplement with herbs, vitamins, and nutrients that helps you with focus and concentration.”

  3. Dave Ruddell says:

    So wait, does this mean playing Brain Age won’t make me smarter?

  4. Robin says:

    Another use of those types of websites/puzzles/brain games are to battle cognitive decline in the elderly. Supposedly there is something to this; I found a study a supportive study and have heard a great deal about it in the lay press (but am unaware of the consensus about this):

    http://www.ncbi.nlm.nih.gov/pubmed/17179457

    My Dad plays TF2, he tells my mom it’s good for him!

  5. chaos4zap says:

    When I was approaching the age where I could legally go hunting with my grandfather, I use to play duck hunt on the old NES non-stop with the assumption that it was preparing me for real hunting. Let’s just say, that was certainly not the case.

  6. Scott says:

    Interestingly, one could infer that this suggests that actual GAME games are more effective for the purpose of brain training than products designed for such training. Take THAT, Mom!

  7. Fredeliot2 says:

    There was a recent study that found significant improvement in stroke recovery with patients using the Wii gaming system. While the products may not improve intelligence beyond some genetically determined limit, they may slow the losses associated with age. Some studies have indicated a benefit from crossword puzzles and ballroom dancing but others have found no benefit. We have a lot to learn about learning. I just hope that they come up with something soon enough to do me some good.

  8. cervantes says:

    This was done with adults. I think it not just likely, but pretty well established, that children’s cognitive chops get stronger if they are intellectually stimulated and challenged. By the time we’re 39 though, we’ve pretty much got to live with what we have.

    On the other hand, it may well be that this kind of training and slow cognitive decline in aging. That’s the other side of this coin and it’s worth testing, I think.

  9. platowannabe says:

    I was disappointed the not see that they also tested the Dual N-Back training. From what I have read, it one seems to have some potential benefits(1). Also there is a study that that seemed to showed ” ’14 hours of training over 5 weeks’ led to measurable density changes for cortical dopamine neuroreceptors.[2]” (source of source for this is wikipedia).

    I guess there will always be ‘But this one really does work’ games out there, but since preliminary studies have been so positive, be nice to know conclusively.

    I had done it for about 5 weeks, maybe 7 hours a week, and I did feel an improvement regarding my ability to focus and concentrate with background noise. Perhaps totally placebo, so it be nice to know for sure.

    1- https://docs.google.com/viewer?url=http://www.iapsych.com/articles/jaeggi2008.pdf

    2- ^ “Changes in Cortical Dopamine D1 Receptor Binding Associated with Cognitive Training”, Fiona McNab, Andrea Varrone, Lars Farde, Aurelija Jucaite, Paulina Bystritsky, Hans Forssberg, Torkel Klingberg. Science 6 February 2009: Vol. 323. no. 5915, pp. 800 – 802

  10. platowannabe says:

    meant 7 hour total over the 5 weeks for me. Apparently I need to start training again..

  11. windriven says:

    “They randomized 4,678 subjects to three arms – one group were trained on programs designed to enhance reasoning, planning, and problem solving. The second group were trained on memory, attention, visuospatial processing, and mathematical calculations. The third group, the control, were given five obscure knowledge questions and asked to use the internet to find the answers.”

    After reflecting on this a bit, I’m not sure I believe the ‘control’ group to have been an appropriate control. Why weren’t the control group simply allowed to watch commercial television? The fundamental question seems to be: is the active mind more robust than the sedentary mind? If so the next question becomes: what is the (mathematically speaking) function that describes the general relationship between stimulation and performance improvement (for some quite specific task, i.e. short term memory or analyzing spatial relationships). One suspects from the discussion above an asymptotic curve but one doesn’t know until the experiments are done. And presuming that the curve is asymptotic, the question becomes: what are the limiting factors?

    In short, ‘can the brain be trained using currently available training products’ seems to be quite too broad a question.

  12. Scott says:

    There was a recent study that found significant improvement in stroke recovery with patients using the Wii gaming system.

    The Wii’s a poor choice to study this, I’d say, because the extent to which Wii gameplay represents *exercise* can be significant.

  13. LovleAnjel says:

    A better control would have been playing a video game that is not designed as “brain exercise”, plus another control group of people doing crosswords/sudoku.

    Anecdotally, playing “normal” videogames vastly improved my hand-eye coordination and response time. Spending 4+ hours a day on Sonic the Hedgehog got me to the #2 spot in response time in 7th grade (just behind the guy who played 6+ hours a day). I was still last picked for softball.

  14. ceekay says:

    I don’t think 10 minutes of training 3X week is enough to improve cognitive function.

    Obviously, they had to dumb-down the time requirements in order to recruit thousands of subjects…. FAIL

    This study is not the last word (and was not well designed)

  15. ceekay says:

    Sorry meant to say

    I don’t think 10 minutes of training 3X week for up to 6 weeks is enough to improve cognitive function.

  16. DevoutCatalyst says:

    Who is the intended consumer for such? Aren’t the gamer, the bibliophile, the oenophile already well served by their interests?What is the value of some additional product for the person who already does mind bending things, be it polyhedral combinatorics, naval aviation, celery breeding, whatever?

    Surf’s up, life beckons.

  17. I know these products work! I have been working with one that improves thought-processing and word retrieval, but I can’t remember the program’s name. :)

    Seriously, I would love to find a game that improves word retrieval. In the last few years that “it’s on the tip of my tongue” feeling has become more familiar.

  18. daniel says:

    My mental model (no pun intended) was that learning anything is good. Within the past two years there was a Scientific American article that neurons need to be engaged within a small window about 2 weeks after they are “born” or else they die off.

    This doesn’t help you learn what you are currently learning, but stimulates neurons for the future.

    Video games often involve significant learning, even if of insignificant things.

  19. antipodean says:

    CLINICALLY PROVEN

    Which is funny when you are a scientist and know that ‘clinical’ mostly means flying by the seat of your pants with the current limited information…

  20. cloudskimmer says:

    Dr. Novella: I noticed you used the term “multitasking.” I’m still confused about that; I thought that the evidence shows that there is no such thing as multitasking, i.e. doing two things at once; all that really happens is rapid switching between the two tasks. Perhaps there’s no real difference, but didn’t recent evidence also indicate that those who thought they were best at multitasking were actually the worst?

    Sadly my local NPR station has been shilling for the ‘brain gym’ during pledge breaks, repeating the claims of the marketers and saying, “I tried it and it worked for me.”

    And I’d like to know if there’s any evidence this can slow progression of dementia or Alzheimers. My brother has my Dad doing some kind of computer program/game to try this. I guess the rationale is that we can’t wait for the results, so we might as well try something. And if Dad enjoys playing, that’s probably a good thing…

    I just wish the marketers would have to provide data to support their claims of improved brain function. Improved over what? And how does it transfer to the real world?

    micheleinmichigan: thanks for making me laugh!

  21. cloudskimmer says:

    Correction: it is “PositScience” which sells three programs, one of which will improve your driving! When I clicked on the link about scientific support, it showed the page as unavailable… little surprise there.

  22. squirrelelite says:

    A couple little anecdotes that may be relevant and at least involve electronic games:

    @michelleinmichigan,

    That “tip of the tongue” feeling sounds familiar. Last summer I had a day time blackout seizure episode. When I came out of it and my son drove me to the emergency room, I was still so befuddled that I don’t think I could even come up with my name or any of the other answers to the questions of the check-in person. Fortunately, my son knew the answers and I told them just to let him answer for me. I remember having trouble maintaining my concentration long enough to play solitaire on my cell phone, which normally only took about 3 or 4 minutes.

    For a couple months after that, I was more functional and had a lot of experience with that feeling. I seriously wondered if I might have suffered some permanent loss as a result. But, fortunately I am pretty much back to normal, whatever that is, and the tip of the tongue instances are much less frequent. Although, I did have trouble remembering Brooke Shields’ name a few days ago.

    Second, many years ago when I was in college, Evelyn Woods Reading Dynamics was all the rage and I read a little about it. It basically consisted of two tricks. One was not sub-vocalizing words as your read because this slows you down. The other was widening your focus and scanning a whole line at a time instead of individual words. I sort of figured out how to do it, but I mostly gave it up. For serious studying, I need to think about what I am reading as I read it and the speed advantage doesn’t help. For casual enjoyment reading, I missed the experience of telling the story to myself as I read it. So, I preferred to take my time and not get it over with too quickly.

    Still, the skills sometimes come in handy. Now, when I play Sudoko on my cell phone I am playing against the clock, not just trying to logically figure it out as I do with paper and pencil. That makes the game a bit different and it is often helpful to scan all the numbers in a square, row or column faster because I don’t individually count them off. Then, I can make a quick “guess” as opposed to a carefully reasoned decision and mostly I get it right. At least I like my times.

    So, I think skills learned in a game or other activity can transfer to and be useful in other activities. But, I doubt if they have any major benefits to your real intelligence.

    Also, I would at least like to think, and there may be some study data to support it, that keeping your mind active whether by playing games, reading or studying, or just chatting on blog-threads helps maintain your longterm mental functionality.

  23. orange lantern says:

    My video game habit is not making me smarter? I refuse to believe your tobacco science!

  24. GeoffreyCoe says:

    This is a really interesting large study and the results and conclusions have to be taken seriously. It is good example of the commercial sector creating products that are “evidence-tinged” rather than evidence-based – actually quite plausible, rides on the general literature of neuorplasticitiy and learning but if you are earnest about improving your cognitive abilities until these computer packages are really tested there is no point in buying them.

    It is also great because it opens up some really interesting questions and many of these are raised in the discussion section of the study. I wondered could the games not be task- specific enough? Generalisation is thorny thing in many aspects of learning and the failure to make the treatment task specific enough to is often a key.

    The other thing I thought is worth saying we shouldn’t make any generalisations how a population in rehabilitation might do. These subjects were drawn from a healthy population. Responses could be completely different in people with cognitive impairments. People with mild cognitive impairment, early stage dementia or TBI may respond quite differently? A crude analogy might be in the gym when using weights. The healthy has a much harder time of improving their strength than those who are weak but have potential for improvement. If you are starting from a lower level of ability you can make much greater gains proportional to your starting ability.

  25. DevoutCatalyst says:

    “…Responses could be completely different in people with cognitive impairments. People with mild cognitive impairment, early stage dementia or TBI may respond quite differently?…”

    Yes, so why this product? As a tip of the iceberg, for example, I’ve worked with people with Down Syndrome, and I’ve seen a child with DS enthusiastically reading at age 4 1/2, another young child who is bilingual, and yet another, a teenager with Down Syndrome, riding his Jet Ski independently, playing 4 musical instruments by the time he reached his senior year in high school, a young man who also found time for devotion to and accomplishments in Tae Kwan Do. How can this software compete with real life?

    Any such brain training software would have to be compelling and rewarding in and of itself — a part of real life — otherwise I’d think you’d be shortchanging people, wasting time jumping folks through hoops instead of enabling them to make leaps by harnessing their own volition. People with cognitive impairment need to get a life, and at least some people in cognitive decline maybe should have been taught how to better pursue their own cognitive self-interests during their lifetimes.

  26. dcelliott99 says:

    There is a big difference between serious brain training programs based on peer reviewed research and casual brain games that have no scientific validation. Frankly, the market hasn’t done a very good job of explaining what a real brain training program is, including the need for scheduled blocks of training time. Here is a site that helps to separate validated brain training programs from casual brain games : http://www.braingamereview.com

  27. Any thoughts on this?
    http://www.madinamerica.com/madinamerica.com/Home/236D66EE-A308-4D89-A4EF-03B868C61B58.html

    A journalist defending his thesis that psychiatric medication doesn’t improve long-term outcomes.

  28. weing says:

    I haven’t read the book. My biased opinion on this is that you get what you pay for. Pay people for being disabled, and you will get more disabled people. I’ve seen enough car accident cases that do not want to get better all in order to get a better settlement.

  29. weing,

    I haven’t read the book either. I’d be interested in knowing whether he’s talking about something real or something he’s invented, and if it’s real, what the consensus explanation is.

    As someone who did poorly for twenty unmedicated years and comparatively well for the subsequent thirteen medicated ones, I have a personal stake in what the science says about long-term outcomes.

    I think you’re saying that we’re seeing correlation, not causation; that disability payments and relatively safe psychiatric medication became available at the same time, and it’s the first, not the second, that’s driving up the rates of people on disability due to mental illness.

    You are certainly correct that we get what we pay for. I don’t mind paying for it, though. My brother is on disability payments for his schizophrenia. He has a subsidized apartment that he has to re-qualify for every five years: he doesn’t automatically lose it if he does well for six months and is able to find and keep a job. He’s also able to work and earn money, which he does. He declares his income every week and gets to keep a certain amount before some of it is taken off his cheque. I think this is a humane and appropriate approach, taking his limitations into account but rewarding him for pushing them. Given the choice between enabling someone less marginal than my brother to work less than they theoretically could, and leaving my brother to live in homeless shelters (which he did for a summer) or spend much of his life in prison (which he was on his way to doing), I happily choose the former. Given the natural history of schizophrenia, I’m interested in the long-term outcomes for my brother with and without medication.

    I see explanations beyond the economic as well. An aging population presumably results in increased rates of disability for all causes. An increase in population overall results in higher absolute numbers. Cultural changes have presumably resulted in people with general malaise accepting disability status on the basis of a psychiatric diagnosis instead of back pain, bringing in our old friend diagnostic substitution.

    But this is all speculation. I’m afraid if I read the book I won’t be familiar enough with the science to know when he’s making shit up.

Comments are closed.