A/B Testing in Education

Back on April 20 of this year, I wrote a detailed explanation of experiments. Because we so often see in the press “breaking news” of the “latest study,” it’s important for critical thinkers to have a good understanding of what makes a study valid. Sample size and composition are important. Many press releases are based on less than 100 observations from a narrow population but reported as if they apply to everyone. 

Another major flaw is that when the scientists or doctors report a correlation, a strong relationship between a drug or practice and an outcome, the news reports it as if there is a cause/effect relationship, which is much less likely and harder to prove. That’s why the most common phrase at the conclusion of these studies is that more research is needed, but this may be an afterthought in a news report if it's mentioned at all.

I read lately about a slightly different type of experiment. It is a natural experiment and has become more common in this era of big data. Instead of inviting participants into a lab, dividing them into two groups, giving a treatment to one group and leaving the other alone and  then comparing results; a natural experiment sets up different groups in real life by feeding them different information and measuring the reaction.

Google, Facebook, major advertisers and others with access to a huge population of followers can vary their messages and compare results. They don’t need volunteers, users of the sites become unknowing subjects.

An ad to one group may have a blue background and the other a green background. Which gets the most clicks or likes or referrals or sales? An advertiser may use different pictures or wording in two different regions of the country. Which gets a better response? Someone calls the number and asks for Mary, not knowing that Mary is a code for a particular radio or TV ad in Chicago as opposed to a different one there or somewhere else. This practice is called A/B testing, and it’s happening constantly, especially on the major social media platforms. What are the best colors, pictures, format, headline wording, etc. to get clicks, shares, donations or return visits? 

It sounds like sophisticated manipulation with no apologies – psychological warfare and an invasion of privacy. But there may be several positive uses for it. I thought of this while watching the Jeopardy! Teachers Tournament last week.

According to Everybody Lies, a book about big data applications, a company called EDUSTAR that makes educational software for kids did a kind of A/B test that yielded surprising results. “One lesson plan that many educators were very excited about included software that utilized games” to teach fractions. They found out that a more standard approach, not using games, yielded better student understanding.

On Jeopardy! right after the first commercial break, the teachers sometimes respond to questions by describing their methods in the classroom, and each seems to takes a different approach. They have their little tricks and games, and they are doing what they are comfortable with or what has perhaps worked for them in the past. These ideas seem to get an enthusiastic reception, but who knows if they really work or which is better for producing better-educated kids – presumably the point of our schools.

They can’t all be right about getting the best outcome. This seems like a perfect setting for a similar kind of A/B testing by increasing the sample size from one to many classrooms distributed over many different schools. Maybe some techniques work better than others. It would be interesting to find out. Of course some teachers would be uncomfortable with this, but surely the objective of education is to make kids more competent, not to make teachers comfortable. Just a thought.