Mark Twain popularized the phrase, “There are three kinds of lies: lies, damned lies, and polls and surveys.” (He may have said “statistics” at the end, but I think this version works as well.)
A new Harris Poll on “alternative medicine” nicely demonstrates some of the problems with polls. The biggest problem is how you frame the questions. You can dramatically affect the results of the poll by exactly how a question is phrased, which other questions come before or after the question, and the overall context of the poll.
In essence a poll is a two-way form of communication. While it is meant to derive information from the subjects of the poll, those subjects are also receiving information from the poll itself. That is the very reason for “push polling” – which is the practice of disguising campaigning messages as a poll. (“On a scale from 1-5, how much are you bothered by the fact that candidate X is a misogynist?”). Yes, Prime Minister has an amusing example of this practice in action.
Interpreting the results of a poll is also not straightforward. You have to know how the subjects of the poll interpreted the questions, and what factors might affect their response. (more…)