A/B Testing in Education

Back on April 20 of this year, I wrote a detailed explanation of experiments. Because we so often see in the press “breaking news” of the “latest study,” it’s important for critical thinkers to have a good understanding of what makes a study valid. Sample size and composition are important. Many press releases are based on less than 100 observations from a narrow population but reported as if they apply to everyone. 

Another major flaw is that when the scientists or doctors report a correlation, a strong relationship between a drug or practice and an outcome, the news reports it as if there is a cause/effect relationship, which is much less likely and harder to prove. That’s why the most common phrase at the conclusion of these studies is that more research is needed, but this may be an afterthought in a news report if it's mentioned at all.

I read lately about a slightly different type of experiment. It is a natural experiment and has become more common in this era of big data. Instead of inviting participants into a lab, dividing them into two groups, giving a treatment to one group and leaving the other alone and  then comparing results; a natural experiment sets up different groups in real life by feeding them different information and measuring the reaction.

Google, Facebook, major advertisers and others with access to a huge population of followers can vary their messages and compare results. They don’t need volunteers, users of the sites become unknowing subjects.

An ad to one group may have a blue background and the other a green background. Which gets the most clicks or likes or referrals or sales? An advertiser may use different pictures or wording in two different regions of the country. Which gets a better response? Someone calls the number and asks for Mary, not knowing that Mary is a code for a particular radio or TV ad in Chicago as opposed to a different one there or somewhere else. This practice is called A/B testing, and it’s happening constantly, especially on the major social media platforms. What are the best colors, pictures, format, headline wording, etc. to get clicks, shares, donations or return visits? 

It sounds like sophisticated manipulation with no apologies – psychological warfare and an invasion of privacy. But there may be several positive uses for it. I thought of this while watching the Jeopardy! Teachers Tournament last week.

According to Everybody Lies, a book about big data applications, a company called EDUSTAR that makes educational software for kids did a kind of A/B test that yielded surprising results. “One lesson plan that many educators were very excited about included software that utilized games” to teach fractions. They found out that a more standard approach, not using games, yielded better student understanding.

On Jeopardy! right after the first commercial break, the teachers sometimes respond to questions by describing their methods in the classroom, and each seems to takes a different approach. They have their little tricks and games, and they are doing what they are comfortable with or what has perhaps worked for them in the past. These ideas seem to get an enthusiastic reception, but who knows if they really work or which is better for producing better-educated kids – presumably the point of our schools.

They can’t all be right about getting the best outcome. This seems like a perfect setting for a similar kind of A/B testing by increasing the sample size from one to many classrooms distributed over many different schools. Maybe some techniques work better than others. It would be interesting to find out. Of course some teachers would be uncomfortable with this, but surely the objective of education is to make kids more competent, not to make teachers comfortable. Just a thought.

Someone is Always Watching

While no one is paying attention, our privacy slowly slips away. We have at our fingertips all kinds of new technology that makes life so much easier and more convenient. At the same time, it puts everyone in a position to be held accountable for every mistake they make.

This was brought home to me by two stories on the same day this week on CBS This Morning. The first was about new technology in cars, and the second about a controversial photograph.

The auto technology story is very interesting. The new touch screens on some of the higher end cars help with navigation for regular trips as well as for emergencies. If the car is low on gas, it will direct the driver to the nearest service station and allow payment from inside the car and allow occupants of the car to order food from the snack bar inside. The Wi-Fi modem also allows drivers to make reservations on the way to a destination.

This sounds very convenient, but all that convenience comes at a price. While drivers take advantage of these new gadgets, the car is recording all kinds of data with “the power of 20 personal computers and [it] can process up to 25 gigs of data every hour – some of it beamed back” to the manufacturer. 

The data includes not only driving habits, but also personal preferences. There is so much data that experts estimate that the data stream will soon be more profitable to carmakers than the car itself. Information about purchases can be sold to advertisers who may also pay to put promotions on the screen. Drivers can now voluntarily share data with insurance companies to take advantage of good driving habits and secure a lower rate.

If people don’t fight to keep this information voluntary, however, think of how fast it can get out of control. This, like the cell phone, is one more source for law enforcement in case of crashes, other motor vehicle incidents or criminal investigations. It is a possible source for insurance investigations or rate setting, an additional source for spam (disguised as targeted advertising) and many other problematic scenarios as the cars record location, stops, speed, erratic driving, hard-braking and driver eyelid movements with inside cameras. 

All this seems harmless and some of it is even good, leading to better safety and maintenance, but make one mistake…. Your car is spying on you.

But spy-like technology is not only in your car: it’s everywhere. The second story was about the prom photo from Baraboo, Wisconsin where a group of 30 high school boys seemed to be making a Nazi salute. The staged photo from last May was posted to Twitter last Sunday and has sparked “international outrage.” 

CBS didn’t identify who posted it or find out why it would be posted six months after the fact. According to one of the students who didn’t participate, the photographer only said, “Raise your hand,” and he thought the idea was just a joke on the part of some his classmates. The photographer denies the charge against the boys saying it was supposed to show them waving to their parents. No one talked to any of the other students to find out their intentions. But it’s too late, as our wonderful technology instantly allowed people around the world to be shocked and outraged.

It is without a doubt that CBS and other news outlets picked up on this story for the shock value. But is it really shocking that a bunch of high school boys would participate in this kind of tasteless, insensitive horseplay? And it wasn’t even premeditated, as was the case of Prince Harry wearing a Nazi uniform to a party when he was two or three years older than these kids. But the outrage merchants never cut anybody any slack. (It’s a good bet none of these boys will be eligible to serve on the US Supreme Court forty years from now.)

Remember, no one is allowed to make mistakes (or bad jokes) anymore. Someone or something is always watching, recording or posting, eager to spread or sell the information. Scary!