Surveys and Critical Thinking

The news media seem obsessed with survey results, people telling us what they think or how they feel.   Being a critical thinker (at least most of the time), I take the results of these self-reporting surveys with a grain of salt.

I am skeptical for several reasons.  It is easy for organizations to word survey questions with the intent of influencing the answer.  People often want to impress the survey taker.  Most surveys leave no room for follow-up or clarifying questions.  The sample surveyed is often done with convenience in mind or is otherwise not properly designed.

I have seen survey questions that are preceded by a paragraph about how terrible or how wonderful a particular practice or policy is.  They then ask if you favor continuing the policy.  It’s important to see the entire question before judging the validity of the answer and brief news stories often omit this information.  In a similar vein, how many times has a salesperson told you directly that anything less than “completely satisfied” on the follow-up survey will reflect badly on the store?

Sometimes people just want to show the survey takers how cool or clever they are.  They try to guess at the desired answer or try to position themselves in the right group.  There are many examples where married couples are questioned separately about something they do together:  housework, managing the finances, general decision making, who disciplines the children or frequency of sexual activity.  Invariably discrepancies appear between the answers of husbands and wives, discrepancies that can be explained only by a misperception or an attempt to give a favorable rather than an accurate answer.  Apply this to teens self-reporting on tobacco or drug use.  Are they being honest or exaggerating?

Some surveys give a choice of answers, but perhaps none fits the exact situation.  I have received many customer service surveys that leave me scratching my head because I did not experience the service they ask about and there is no “Not Applicable” option.  I can’t give them the correct answer and there is no way to clarify.  The same can be true with public opinion questions.  My “10” or “completely agree” may differ, not from your experience, but from your definition of these terms.

Finally, there are people who, either purposely to influence the outcome or through negligence, survey an incomplete or biased sample.  I sometimes get unsolicited phone surveys, which I hang up on.  I wonder what they do next.  Do they get someone else from their list with the same demographics (and how do they know my demographics anyway?) or more likely, do they just go to the next name on the list?  Sometimes they survey likely voters and sometimes just registered voters.  It makes a difference.

So I take these reports in the news less seriously.  When I hear that consumer confidence is slipping, it’s interesting, but not as reliable as the retail sales numbers at the end of the month.  What people say and what they do often differ.