Flashback - Diet Soda and Depression

[I wrote this over seven years ago about another meaningless study meant to make disturbing headlines for the news media and possibly to attract more funding for the researchers. The only party left out of the equation, as usual, was we, the consumers.] 

I have mentioned before the tendency of journalists and the media to try to get and hold our attention with emotionally charged pictures and stories. We regularly see news about the results of studies that seem surprising, scary, or shocking, but they are often somewhat meaningless. They challenge our ability to remain calm and think rationally about the subject. Here is another example.

“A new study finds that people who drink diet sodas or fruit drinks are more likely to be diagnosed with depression.” The article continues very responsibly to explain the size and nature of the study and to emphasize that a link does not necessarily mean that drinking diet sodas causes depression, but the headline - "Drinking diet soda linked to depression" – has us hooked. (With sugary soft drinks being blamed for obesity - blame the soda, not the person drinking it - our choices are narrowing.)

Thinking critically about it, we know that correlation is not the same as causation. "Linked to" isn't the same as "caused by." We may wonder what we are supposed to do with this information – stop drinking diet soda so to avoid depression or start drinking lots of coffee, which the article tells us may have the opposite effect? I don’t think it works that way.

Later in the article they say, “more research is needed.” So what was the point?

Why should we even care about these kinds of studies that give preliminary findings, or publicize findings before they are presented for formal review, or rely heavily on self-reporting as opposed to objective observation?  As I pointed out before, we don’t have the time or energy to be worried about everything, so there is nothing really useful about such news. But it does make for catchy headlines.

[After all this time nothing has changed. Basically meaningless preliminary studies continue to be published and highlighted in the news almost daily, and it won’t stop until a large number of Americans wise up to the tactics.

On second thought, one thing has changed. Judging from comments on social media, some people do have time to worry and express an opinion about everything, but that's another topic. ]

Understanding Experiments

When most people hear the word experiment, they picture a scientist in a lab coat with bubbling beakers and test tubes of mysterious liquids. This is not the real meaning. As a result of this misconception, news of the latest study is usually misleading.

An experiment is a rigorous process of testing an idea with the intention of either solving a problem or of improving a situation. Necessary first steps include defining the problem to be solved or determining how results will be measured. No one can claim improvement without measurements to compare (before and after). In the first case, the experiment is successful if the problem goes away. In the second, success is judged by the measurable amount of improvement.

When a well-designed experiment is successful, the implication is that the same solution can be applied widely to other situations: to solve the same problem elsewhere or to achieve the same amount of improvement. (Usually others replicate the experiment before findings are accepted.)

To achieve satisfactory results, any experiment must be well designed. Sloppy studies lead to problematic conclusions, ones that can’t be counted on to solve anything. Even the best experiments can yield bad information just based on the amount of diversity in the world and the fact that fluky things happen. 

Many experiments or studies the public is exposed to in the media relate to drugs or other remedies, and a strict procedure must be followed to ensure the conclusions are valid, otherwise the drugs or other remedies get on the market without proof that they are safe and effective.

In drug studies researchers try to choose a sufficiently large sample, because the larger the sample, the lower the chances of getting some oddball results just because you happened upon some atypical participants. A bigger sample tends to average out any unusual individual readings. 

The next step is to divide the sample into two parts that look as much alike as possible: same proportion by sex or race or education or income or background or location or any other feature that might affect the results. Sometimes this can only be done by random assignment to one group or the other. It's better if researchers have a good understanding of all the characteristics that might influence results and can make the two groups look as much alike as possible relative to those characteristics.

One group is treated – given the pill or the information or other treatment – the other, the control group, is given a fake pill (placebo) a sham treatment or left alone.

Afterward the groups are compared statistically to see if the change (hopefully an improvement) in the first group is significantly better than the change in the second. (Yes, there is often a change in the control group merely because the know they are participating in the experiment and believe the placebo is a real remedy.)

Ideally the people doing the testing are not aware of who is in which group (double-blind).

We know from the news that even these careful experiments can go wrong. Sometimes drugs are withdrawn from the market due to problems discovered only after they are released, released to a much larger sample size. That is why I have come down so hard and so often on vitamins and other dietary supplements where the law exempts them from the need for any research at all to prove safety and effectiveness. (They rely on endorsement, not proof, and use weasel words to imply effectiveness.)

Many times in the past I have also criticized experiments because they have been sloppy about their design. Sometimes they don’t define the problem until after the test is done and announce the findings with a press release. Sometimes the samples are too small due to budget constraints or laziness. Sometimes they rely on self-reporting so there is no real measurement. 

Finally businesses and educators like to say they are experimenting. In both cases it is rarely true. They don't set up two groups to compare. Then they measure by gut-feel.

Businesses are in too much of a hurry to do it right. They just try things, sometimes multiple things at the same time, so who knows which ones have a positive or negative or neutral influence on the final outcome.

In education they are still arguing about measurement. Teachers don’t want to be paid based on test scores, but haven't suggested a more acceptable criterion to measure their results. Yet school systems continue to try new methods and approaches without ever satisfying the first step – how to objectively measure real improvement.

Without understanding experiments, it’s too easy to be fooled by people who don’t know what they are doing and by people who do but are just trying to sell us a bill of goods.