Importance of Clear Definitions

If you are familiar with the movie Ghostbusters (1984), you know about Zener Cards used for ESP testing.  Each card has one of five different patterns on the front (cross, star, square, circle, vertical lines) and the person being tested, the subject, is asked to identify the pattern while the tester looks at each card.  To determine if the subject has ESP, the tester is looking for a sufficient number of correct answers to rule out guessing.  Anyone should be able to guess around 20%, one out of five, consistently without any special powers.  Then again, anyone could role five sevens in a row with a pair of dice if you allowed enough tries – or it may happen on the first five tosses just by chance.  That’s why ESP experiments must be large and well controlled.

In the 1930s Dr. S.G. Soal, a British parapsychologist set up such a test.  “From 1936-1941, he performed over 120,000 trials of card-guessing with 160 participants without ever being able to report a significant finding.”  This turned him sour on ESP until a colleague suggested he review the data and see if, instead of correctly identifying the target card, anyone identified the card turned up before or after it.  Now he was getting somewhere!  He published his findings but years later they were challenged on the basis of errors and fraud in the data.

The point is not whether ESP is valid or not, but rather how important it is to decide what you are measuring before you measure it.  If what you are measuring isn’t clear to everyone or you decide after the fact to look at the data in a different way, the results will surely be suspect, at best.  If ESP can be identifying the target card or one near it, depending on the results, it may as well not be defined at all.

I think about the need for clear definition when I read news like this from NBC: “Sexual Assaults Increased at Two of the Three Military Academies.”  Coverage of this story was probably motivated by the recent Marines’ social media scandal. The article shows a graph of reported sexual assaults from each school annually since 2007.  The latest annual increases at the Military and Naval Academies were less than the reduction at the Air Force Academy to bring the total for all three down to 86. 

The article goes on to say:  “Survey responses indicated that 48 percent of female cadets and midshipmen and 12 percent of male cadets and midshipmen experienced sexual assault at their respective academies, according to the report by the DoD Sexual Assault Prevention and Response Office.”  Applying these percentages to the total enrollment by gender works out to 650 sexual assaults per year.  This is far more than the 86 reported above.

So they may have a couple of problems, one with reporting and one with consistent definitions.  It could be that the academies are not getting the compliance with their reporting policy, or it could be that each survey respondent has his or her own definition of what constitutes sexual assault.  This is not uncommon on self-reporting surveys, especially around sensitive subjects, especially in this case where an online search easily turns up multiple definitions.  Indeed this website states that “the exact definitions of the crimes that fall within the category of sexual assault differ from state to state.”

This is far more serious than whether someone has ESP, and far more important to clearly define what we are talking about and to push for conscientious and accurate reporting.