How to Think About Survey Results

The USA Today runs a daily feature called USA snapshots.  They are "easy-to-read statistical graphics that present information on various issues and trends in a visually appealing way," addressing a particular issue in a number of different subject areas including politics, money, sports, news and entertainment.  They sometimes refer to a current news item, movie release or sporting event, but the rest of the time they are just random subjects.  They do provide some good examples for the practice of critical thinking.  (Anyone can review a large number of these simple graphs by searching on the Internet for USA snapshots.)

When looking at these graphs and pie charts or any others it's important to consider a couple of questions.  What type of question is being asked, and what is the source of the data?

One category is facts or statistics.  These are easy to look up and verify.  The source of the information may be government or industry records.  Some typical examples from from the daily inserts:  NCAA Division I all-time scoring leaders, top grossing superhero films, most popular cosmetic surgery and soft drink consumption trends.  The first two are straight out of the record books, so to speak.  The third comes from the American Society for Aesthetic Plastic Surgery and presumably they should know.  The soft drink information comes from "Beverage Digest," another reasonable source.

Another category relates to choices or behavior.  They ask people whether they have ever come in late to work or left early knowing that the boss was out of the office.  The sample of 507 workers is not particularly large, but they should know if they have ever done it and may be willing to honestly confess to it.  It is a little puzzling, though, that the pie chart shows 3% who answered "Don't Know."  A question for 1383 adults who have had allergies for more than 3 years asks whether they stick to the same brand of allergy medicine or tend to switch around.  The primary problem with behavior questions like these is that they are self-reporting.  There is no safeguard against lying or  exaggerating.  This is especially a problem as questions get more sensitive and personal, such as the use of drugs, criminal behavior, smoking, drinking or sexual practices.  The above questions seem innocent enough that people wouldn't be inclined to falsify and even if they did, the  consequences of a bad survey result are small.

Other surveys deal strictly with opinions.  Like the behavior category the self-reported nature of the answers is always problematic in that they are not verifiable.  We must take the word of the individuals.  That is something always to keep in mind.  The graphs in this category ask questions like how many episodes of binge TV viewing are too many - asked of 800 people who admitted to watching at least 3 TV episodes in one sitting.   PETA asked 1,234 New York City voters if they favored or opposed the banning of horse-drawn carriages in the city.  Be careful of this result since we don't know how the question was couched, that is, what questions were asked to lead up to it (setting the mood) and how exactly was it worded?  Because it was a PETA survey, it is reasonable to suspect that they favor a particular answer.  (FYI, most opposed the ban.)

Finally, there is a category that seems like an opinion, but it's hard to determine the basis for any opinion on the subject.  A LoanDepot survey of 1000 adults asked if it was easier or harder to get a mortgage today than it was a year ago.  Do people apply for mortgages every year?  That seems unlikely.  It doesn't even specify that these adults were in the process of applying or had applied in the recent  past.  Who are these people and where are they getting their information that forms the basis of such an opinion?  Yet, more than half (78%) had an opinion while 22% answered, "not sure."  This is truly puzzling and perhaps disturbing that so many people form opinions on "gut feel" or, believing that they should have an opinion,just make one up.  On this subject it's harmless enough, but are there implications for bigger issues?  Likewise, it's a problem that the organization doing the survey and the paper printing it don't question it.  How many people would be inclined to form their own  opinion,  based on the opinions of others that were based, in turn, on nothing substantial?  In a democracy where lawmakers follow poll results so closely, it's scary to think about.