Articles

Scientific American Mind Is Not So Scientific

When Scientific American first announced that they would publish Scientific American Mind, I hurried to subscribe, thinking it would keep me informed about new developments in a field I am passionately interested in. I have enjoyed the magazine, particularly the regular columns, the news items about research findings, the reviews that alert me to books I will want to read, the “Ask the Brains” Q and A, the challenging “Head Games” quiz, and the presentation of many intriguing ideas. The board of advisers is impressive, and the columns by Christof Koch, Scott Lilienfeld, Hal Arkowitz, the Ramachandrans and others have been consistently excellent. Unfortunately, some of the other articles have descended into pop psychology, speculation, poor science and even pseudoscience. Contributing editor Robert Epstein’s articles have particularly raised my blood pressure.

Love-Building Exercises

In December 2009 I was annoyed enough to write this letter to the editor:

After reading Robert Epstein’s article in the last issue, I had to go back to the cover and verify that the word “scientific” was indeed part of the title of your magazine. The Love Building Exercises he recommends are more appropriate to a magazine of fantasy and science fiction.

Two as One — feeling that the two of you have merged?
Soul Gazing — looking into the very core of your beings?
A Mind-Reading Game — wordlessly trying to broadcast a thought to another person?
Love Aura — feeling “eerie kinds of sparks” when your palm is close to another’s?

Thought transfer? Auras? Come on! Shame on you for publishing such metaphysical pseudoscientific psychobabble!

They published my letter to the editor with the heading “Hating ‘Love’.” There was no response from the author.

 Are You Mentally Healthy?

In a March, 2010, article, “Are You Mentally Healthy?”  Epstein presented a screening test that he had developed for mental health disorders and named after himself. He thought his test was more reliable than any of the other tests he found on the Internet because those other tests had not been scientifically validated. His “validation” consisted of his own findings that scores on his test predicted seven important factors related to mental health, such as whether they were employed, how highly they rated their personal and professional success, and whether they had ever been in therapy. (John Nash had been treated for his schizophrenia, but he was employed, won a Nobel prize, and had lasting personal relationships. One wonders how he would have done on Epstein’s test.) Essentially Epstein tried to defend one unvalidated test by showing that it correlates with another unvalidated list of factors.

For fun, I took Epstein’s test. I could see that most of the questions were designed to elicit specific symptoms of depression, mania, anxiety, compulsions, etc. and it appeared to be little more than part of a checklist that a psychiatrist might use to remind him of questions to ask in taking a conventional psychiatric history.

Epstein has not tested people who are known to have mental illness and people who are known to be mentally healthy, but only random people who found his questionnaire on the Internet. He has not defined mental health, much less measured it.  He has only shown that his test scores predict a person’s answers to specific questions that are part of the test itself, questions that he personally thinks are related to mental health.  People with mental illnesses may not answer the way he thinks that they will.  And on the other hand, mentally healthy people might answer the way he thinks only mentally ill people would.  The only way to be sure that a survey works is to “test the test”: to see if mentally ill people actually score high on the test.  Epstein hasn’t done that.

Any test, questionnaire or instrument must be checked for both reliability and validity: reliability means it will give consistent, reproducible results, and validity means it has been compared to some other standard to ensure that it is actually measuring what it claims to measure. For instance, a new kind of thermometer might reliably give the same result every time, but the readings wouldn’t be valid for diagnosing a fever unless they agreed with the readings on a mercury thermometer. A stopped watch reliably shows the same time each time it is consulted, but it is not valid for telling the time.   Epstein may think he has “validated” his questionnaire, but he hasn’t.” He has not shown that the test has any validity for predicting the presence of mental illness.

I tried checking none of the items, and it told me

You haven’t checked off any items, which suggests that your mental health is excellent. If you still have concerns, you can find qualified, licensed counselors and therapists at websites such as Find-a-Therapist.com, GoodTherapy.org, NetworkTherapy.com, PsychologyToday.com, Metanoia.org, AskTheInternetTherapist.com, and TherapistLocator.net. You can also get referrals through your family physician, HMO, or local hospital or clinic. If you are worried that you are losing control of your life, consider taking the test at HowInfantilizedAreYou.com.

 

Then I tried checking all of the items. This time it told me

In some respects [sic] you scored outside the range of functioning that is usually considered normal. This suggests that you should probably [sic] consult with a qualified mental health professional for further testing or treatment. Area(s) of possible concern (expressed in the diagnostic language that will be familiar to your therapist):

Substance Abuse
Psychosis
Depression
Mania
Bipolar Disorder
Mood Disorder
Phobia
Social Phobia
Obsessive-Compulsive Disorder
Posttraumatic Stress Disorder
Generalized Anxiety Disorder
Anxiety Disorder
Relational Disorder
Sexual Disorder
Eating Disorder
Impulse Disorder
Personality Disorder
Somatoform Disorder

You can find qualified, licensed counselors and therapists at websites such as Find-a-Therapist.com, GoodTherapy.org, NetworkTherapy.com, PsychologyToday.com, Metanoia.org, AskTheInternetTherapist.com, and TherapistLocator.net. You can also get referrals through your family physician, HMO, or local hospital or clinic. If you’re worried that you’re losing control of your life, consider taking the test at HowInfantilizedAreYou.com.

Then I tried checking all the odd-numbered items, resulting in a verdict of pretty healthy but having a possible eating disorder, followed by the same canned universal suggestions.

Checking all the even-numbered items gave me a possible diagnosis of social phobia. Just for the halibut, that time I also reported that I was 10 years old and had a doctorate. That kind of question doesn’t go into the scoring, but Dr. Epstein uses it for his research-by-Internet-stealth. Underlining just how unreliable such research is.

How Do You Handle Stress?

Epstein’s latest article is another example unworthy of Scientific American. In the September/October 2011 issue he gives us “Fight the Frazzled Mind: A new study suggests that preventive, proactive approaches are the most helpful — and that our stress management IQ is painfully low.” The new study is one Epstein did himself and presented at a conference but did not publish in a peer-reviewed journal.

The study looked at 3304 subjects who completed an online test. They were asked to rate, on a 10-point scale, how stressed they were, how generally happy they were, and how much success they had had in both their personal and professional lives. The main body of the test involved questions in four areas of competency: manages sources of stress, practices relaxation techniques, manages thoughts, and prevents stress from occurring. As far as I can see, this identification of four competencies and the corresponding questions are nothing but his own invention. Some examples of individual questions that he thinks can be used to measure those competencies:

  • I try to schedule appointments and meetings so that they won’t overlap.
  • I schedule some relaxation time every day
  • I’m aware that my thinking is sometimes unclear or irrational
  • I keep an up-to-date list of things I’m supposed to do.

He says he was surprised by one of his findings: that prevention is by far the most helpful competency when it comes to managing stress. His take-away message is that it is better to avoid stress in the first place than to use techniques like relaxation after stress has developed. I am surprised that he finds this surprising.

He says his study also shows that people who have had training in stress management are better at it, and the greater number of hours of training, the better the skills. He doesn’t explain, elaborate, or quantify. “Stress management training” is not defined.

Then, out of the blue, he offers six strategies he says were “suggested by the new study” to fight stress before it starts:

  • Seek [stressors] and kill
  • Commit to the positive
  • Be your own personal secretary
  • Immunize yourself (through exercise, thought management and relaxation techniques)
  • Make a little plan (for each day)
  • And make a big plan (for the long term)

But these weren’t really suggested by the study, they were strategies that he had already decided ought to be stress-reducing and therefore ought to be included in his questionnaire. Do I smell circular reasoning? And of course, he has no evidence that efforts to adopt these strategies will have any measurable effect.

Then he says the worst news is that on his 100 point scale, people scored an average of 55.3.

If you think of that as a score on an exam at school, that means that on average, people get a grade of F when it comes to managing the inevitable stress they face in their lives.

A Faulty Method

This seems to be Epstein’s modus operandi: he thinks up his own questionnaire to try to measure something, and without even trying to validate it he proceeds to use it in a study, and then gives talks and writes popular articles about his results and gives pop psychology advice allegedly based on the studies. But these studies never get published in peer-reviewed journals and never show up in PubMed. Epstein’s website offers the stress questionnaire and a book of stress relief games, along with tests he has developed to measure such things as “adultness” and “love competency.” No, I’m not making this up! I took the adultness test and found it very entertaining. You might too.  His questionnaires are reminiscent of the kind of questionnaires that are ubiquitous in popular magazines, where your score allegedly predicts whether your marriage is likely to last or tells you whether your self-esteem is high or low.

The flaws of this method are obvious, as can be seen in the new “Stress” article. Questionnaires must be validated before they can be used to measure anything. Terms like “stress” must be objectively defined. Self-reports of stress, happiness and success may not correspond reliably to any quantifiable reality. Subjects who self-report as happy, successful and non-stressed can be expected to answer the questionnaire items from the biased perspective of their self-image. And a score on a made-up test can hardly be compared to an F grade in school.

Conclusion

Epstein is much better at self-promotion than at science. In my opinion, Scientific American Mind would be better off without him. Let this stand as an open letter recommending that they remove him from the position of contributing editor and that if they consider publishing any more of his articles, they first submit them to peer review by rigorously scientific psychologists with good critical thinking skills, such as Scott Lilienfeld, who is already on their board of advisers and is also a fellow of the Committee for Skeptical Inquiry and the editor of The Scientific Review of Mental Health Practice. I’m picking on Epstein as a bad example (and a particularly prolific one), but he is not the only offender. Other similarly questionable articles have slipped past the editors. With a little weeding, Scientific American Mind could be what its name promises: scientific.

Posted in: Neuroscience/Mental Health, Science and the Media

Leave a Comment (16) ↓

16 thoughts on “Scientific American Mind Is Not So Scientific

  1. Jeff Rubinoff says:

    Did Epstein come over from Cosmopolitan?

  2. @Jeff Rubinoff- damn, my thoughts exactly, you beat me to it.

    Those kind of articles are the reason I stopped reading most of the parenting magazines. Lots of scientific sounding articles that are based on the same kind of circular reasoning you describe here. Sad to see it in SA.

  3. aegimius says:

    I can’t say I am surprised. SA may occasionally have some great articles, but it isn’t what it used to be. Although I was only an occasional reader, I could no longer take SA seriously after they promoted a scam company(I mean SCAM, bot sCAM) 2 years ago.

    This is the scam-promoting article by Larry Greenemeier from 2009 – http://www.scientificamerican.com/article.cfm?id=uav-drone-stratosphere.

    I was shocked when I read this. I couldn’t get over Greenemeier’s credulousness, and the incompetence of SA’s editors. How could they let this scam promote itself through SA? I wrote both Greenemeier and the editor and never got a reply.

    The article makes it look like Sanswire is on the cutting edge of airship and telecommunications technology – they aren’t. This is a company that has never had a successful product in all their years of existence, and have never sent anything into the stratosphere or anywhere close. On the other hand, they’ve had very serious legal problems with the SEC, FBI and IRS, resulting in the imprisonment of the former CEO and former CFO. The company was even delisted by the American Stock Exchange in 2006. The overwhelming majority of the company’s revenues were proven to be fraudulent by investigators – http://www.fbi.gov/miami/press-releases/2010/mm072610.htm

    Sanswire is now known as “World Surveillance Group” – http://www.satnews.com/cgi-bin/story.cgi?number=552623191

    It used to be called “GlobeTel” – http://investor.news.com/bnet/news/read?GUID=6669067 when their now imprisoned former CEO Timothy Huff was in charge.

    It is typical of such companies to change their name every few years to attempt to hide their ugly past.

    The company’s press releases are always nothing but hype and very misleading. Seth Jayson from Motley Fool wrote a series of articles that helped expose this company for what it truly is – http://www.fool.com/investing/general/2007/10/12/the-end-of-an-ugly-story.aspx

    Scientific American should be ashamed of themselves, especially the editorial staff. A quick Google search turns up a ton of info that shows what a scam this company is(although the name changes make it a little confusing for those who are not familiar).

    Even though I never heard back from SA, hopefully they will look into the backgrounds of companies they write articles about from now on. It’s the LEAST they could do.

  4. borealys says:

    Thanks for this analysis. I’m also a SciAm Mind subscriber, and also generally enjoy the magazine — but I too sometimes find myself scratching my head over some questionable content.

    I visited Epstein’s website after skimming through his article, and was astonished at how juvenile it felt. My friends and I went through a bit of a phase in our undergrad years where we’d take questionable, unvalidated online tests and email the results to each other for laughs. We never took them all that seriously because most of them were very obviously just meant for fun, like a Cosmo quiz. Those quizzes were what came to mind when I started perusing Epstein’s website. The titles alone… wow. Just wow.

    Coming at it from another angle, I might once have been entertained by those quizzes. But knowing that they’re being represented as something more serious really sucks the fun out of them.

  5. windriven says:

    Echoing aegimius, I have found SA to be of variable quality for some time now; I finally gave up on them about 15 years ago. In fairness their sins have more often been venial than mortal but have been frequent enough that I now buy from time to time off the news stand rather than subscribe.

    There seems to be an erosion of intellectual rigor throughout our society. First rank journals have had to retract studies that clearly should never have been published. Marquee medical schools embrace pre-scientific nonsense. 85 years after Scopes we’re still arguing over teaching creationism in some of our schools.

  6. Did Epstein come over from Cosmopolitan?

    Exactly! This reminds me of WooMD.

    KA

  7. Ha I took the WHOLE adultness quiz. I’m sitting on the toilet so I figured it was a worthwhile activity. Maybe he should have asked “Do you unwind from a hard day at work by sitting on the crapper reading blogs for two hours?”

    Anyway. Wow. Can’t quite figure out what those “questions” say about his view of children OR adults. Maybe he should write scripts for crappy daytime sitcoms. Yah, seriously. He should just go back to Cosmo and Glamour magazine. He’d have a promising career there. They should use his quizzes as an example in an undergraduate “research methods in psychology” course for how NOT to write a survey.

  8. Okay, I couldn’t resist, I also took the adultness quiz. And my response to my results are.

    Na Ner Na Ner Naaah Ner, I know you are but what am I?”

    What silliness.

    As an aside, I hate quizzes that don’t offer honest answers. You’d think he might have considered having a “sometimes” option on many of those questions.

    And how does how well you hear or see factor into whether you are an adult or not?

  9. borealys says:

    In fairness their sins have more often been venial than mortal

    Like loading up the whole thing with ads for wooey crap? Every time I pick up an issue of SA (or any other mainstream science magazine, to be fair), I find myself goggling at the ads. I know that ad revenue is ad revenue, but my idealistic side likes to think that there should be a line when it comes to pseudoscience in a science magazine. Just being in the pages of SA, even if only as a paid ad, lends these things a credibility they don’t deserve.

  10. dchamney1 says:

    More on their advertising policy;
    SciAm, and all other Nature Publishing Group publications, leaves it up to the customer (advertiser) to supply information that is: “true, Accurate, and not misleading and nothing contained in it is liable to bring NPG or the Journal into disrepute;”.

    http://www.nature.com/advertising/resources/tandc.html

    I can only assume that they don’t really care if the advertising brings them into disrepute, because even though the ads cause many if us to question the magazines status as a ‘Scientific’ journal, they’ve chosen to run the misleading ads anyway.

    They may feel that they are indemnified from loss by their advertising policy, but I’m sure that many jurisdictions do not see it that way.

    The reality of it all , at least in my opinion, is that the ads aren’t much worse than the Epstein’s articles at bringing the SciAm family into disrepute.

  11. Prevention is helpful in managing stress?! Mothers already know this, which is why we carry kiddo snacks in our bags. How did he make it to adulthood and not know this?

  12. nitpicking says:

    The September issue of the main Scientific American (the “Cities” theme) is embarrassingly, stupidly lacking in science. I can’t see why I would renew my subscription.

  13. Pingback: eu textile news
  14. bobtheoutcast says:

    That adultness quiz is unbelievable. At first I was amused that so much of it seemed to be rote fact-based knowledge checking (what’s the difference between a psychiatrist and a psychologist? How old must you be to vote?) and a couple inexplicable simple math questions. Is our ability to operate a calculator vital to being adults?

    Where it gets horrifying is when you get to the end, and his write-ups for the categories make it clear that he doesn’t even have the correct answers to all his own fact-based questions. “Adults know that condoms often fail” was the particularly egregious one that drove me to comment.

Comments are closed.