How Science Works

Everyone thinks he understands science:  scientists do experiments and prove or disprove their hypothesis, that is, their guess about how the world works.  Then we see it on the news or in advertisements and follow their advice.  That is how we live healthier and smarter.  But if that is the case, why do we see so much contradictory information?  Unfortunately, science is not nearly that simple.

Let’s talk strictly about health issues.  Scientists do experiments to find out how we can be healthier and how we can avoid things that make us less healthy.  The problem is that everyone’s body is different and may react differently to different substances or remedies.  Scientists must take this into account when designing experiments.  They do so by selecting a large and diverse sample of test subjects, a sample that best represents all of us, so that individual differences cancel out as much as possible.  Still those differences could lead to erroneous results, and they should ask others to try to duplicate and confirm the original findings.

Unfortunately, scientists are human too.  They know that to get recognition and to ensure the continued flow of grant money or corporate support (so they can keep working), they need to hurry findings to press to beat the competition.  The journals that publish scientific results that we hear of on the news should thoroughly review them before publishing, but this doesn’t always happen either.  The publishers also feel the competitive pressure and may cut corners.  What we are left with is preliminary or unconfirmed findings as headlines.

This review process was the subject of a recent PBS interview with a cofounder of Retraction Watch.  (It’s 6 minutes and worth watching.)  They found that in scientific journals there have been “10 times as many retractions…in 2010 as in 2001.”  One reason for the increase may be that retractions are easier to detect with the greater use of computers.  On the other hand, misinformation also spreads so much faster for the same reason.

One reporter took this to the extreme as part of a documentary film meant to point out the flaws in the system.  He did a real study on the effects of chocolate on health.  It was set up properly with a control group and two test groups, but the sample size was very small, only 15, and not representative.  Another flaw was the he tested not for a single benefit but for about a dozen.  With that small of a sample and so many variables, the possibility of meaningful, statistically significant results in at least one area is almost assured.

When he found that those participants who ate dark chocolate as part of their diet had significantly better weight loss (one of the many factors tested for), he looked for someone to publish it.  Needless to say this kind of headline-grabbing news was just what a journal hopes for.  He found a journal that, with minimal due diligence, scheduled it for publication.  (Note that he made no effort to cover up or conceal the shoddy design or lack of review or even that he used a fake name on his report.)  Soon reporters and journalists called, but few asked the right questions or knew the right questions to ask.  News organizations worldwide reacted to a press release and the news appeared in print and on television from Germany to Australia, including in the US.

He followed up with this paper titled, “I Fooled Millions into Thinking Chocolate Helps Weight-loss.”  This is a great read which explains how science should work, but how easy it is to fool publishers and journalists who are under constant pressure to fill print pages and airtime.

This is a good warning for everyone.  Too many news stories are broken prematurely.  Some have the courtesy and true understanding of science to label a study as preliminary, but even when they do so, it’s often presented only as an afterthought.  Scientists are under such pressure to produce results that cutting corners is always a temptation.  And the speed and volume of communications in 2015 is so incredible that misinformation is rampant.  Critical thinking is a must!