Publishing is the currency of academia, and journalists need to develop a set of tools to decipher it. They need to know the difference between dependent and independent variables. They need to understand the half dozen different types of study designs that researchers use.
But the most important thing to keep in mind, according to Michele Walsh: They need to question any claims of causality. Otherwise, they might be tricked into publishing these blockbusters – which demonstrate a correlation but not causation:
- As consumption of high fructose corn syrup drops, so do pedestrian traffic deaths.
- As sales of ice cream go up, so does the murder rate.
- As height increases, so do reading skills.
Walsh is a researcher at the University of Arizona, and an expert on methodological issues and research design. Along with researcher Kara Tanoue, Walsh led journalists at a National Press Foundation obesity training session through an explanation of:
- The different types of studies, from longitudinal to case-controlled to cross-sectional and meta-analyses;
- An explanation of “statistical significance” and whether that means a relationship a study highlights happened by chance; and
- How to interpret “p values,” the commonly cited statistical term that conveys whether something should be considered significant. She cautions that the “number has nothing to do with the importance of what you found.” Instead, it merely means that the event being witnessed wasn’t random.
They also gave tips to reporters for what to do when reporting a journal article on deadline. If the reporter only has 30 minutes, they suggested reading the summary, abstract and conclusion. Many writers ignore conclusions, they said. And the abstract is not enough, as it will rarely hint at limitations.
If a full hour is available, they suggested drilling into the study’s methods, examining who was studied and what was measured. If the study’s authors are available, ask them about the study’s practical impact and whether it is generalizable to a broader population. Then seek out other literature on the topic.
Walsh pointed reporters to other sources:
STATS.org, a collaboration between Sense about Science USA and the American Statistical Association that aims to increase statistical savvy in the media. The group’s advisory board consists of statisticians who are willing to take calls from journalists on deadline.