Scientific Studies

A link to an interesting YouTube segment has been going around.  John Oliver talks about scientific studies in his humorous and irreverent way.  But many of the points he makes should be taken seriously.

Television and FaceBook are filled with studies and many of them seem to contradict each other.  The problems he points out include that scientists are under pressure from their academic institutions or employers, and negative results don’t get published – even if it would be important to know those negative results (to keep from wasting money, for example).  The confusion about which study to take seriously comes from the lack of efforts to replicate previous studies, a very important step in the scientific process.  If a study cannot be replicated by peers, as well as reviewed to ensure proper procedures were followed, its result is not considered valid.  But the pressure to find something new and exciting reduces the availability to get funding for replication studies, meaning those one-time results may have been a statistical fluke rather than a great scientific finding.

To get those all-important, positive, newsworthy results, researchers may resort to various gimmicks, such as using a small sample size, testing for so many variables that at least one will by chance show a statistically significant result, publishing results from lab rat studies as if they are equally valid for humans (which they are not) or publishing a press release with a sexy headline hoping the journalists will not dig too deeply into the substance of the limited experimental findings.  In addition, there are a lot of charlatans on TV and on the Internet describing themselves as scientists and using a lot of scientific-sounding jargon to sell whatever they have to offer.

This is an important subject, and if treating it humorously gets the point across, all the better.

After spending the 19 minutes watching that YouTube version of the Last Week Tonight episode from HBO, I soon saw an example of the problems.  The next day a Health Minute episode on local TV news featured a new study proclaiming that one minute of vigorous exercise was as good as a 45-minute moderate workout.  “Not having the time to exercise is no longer an excuse,” they announced.  I couldn’t find the same piece on the Internet, but instead found this New York Times wellness blog with exactly the same message.  The headline read:  “1 Minute of All-Out Exercise May Have Benefits of 45 Minutes of Moderate Exertion.”

Well, if you read only the headline, you have gotten two things:  another excuse to take it easy and a bunch of bad information.  As it turns out some folks at McMaster University in Canada chose 25 out-of-shape young men, took some biometric data from each and randomly split them into 3 groups:  one to ride moderately on an exercise bike for 45 minutes 3 times a week for 12 weeks, one to ride for only 10 minutes with three 20-second bursts of intense riding, and one to do nothing out of the ordinary (the control group).  They compared readings at the end of the test and found that the physical improvement for groups one and two were comparable and better than the control.

This is good information for those interested in interval training effectiveness, but the sample size was so small (8 per group) that the room for statistical error based on individual differences was huge.  Also, dividing groups randomly is a good default when there is not a better way to do it, but with only 25 people, it shouldn’t have been that difficult to come up with a more careful method.

Without getting too deep into the experimental design, it’s pretty easy to see that this was a small test without much rigor and certainly not what the headlines would lead us to believe.

This is why critical thinking is so important.  The news media, even the reputable NY Times, don’t care so much about the details as they do about catchy headlines to sell newspapers and airtime.  We can easily be led astray.  We can waste money or go down a dead-end path with our lives by putting our faith only in those studies that seem to tell us what we want to believe anyway.  And if you don’t like the study this week, just wait for the next one to come around.