My friend shared a post on my Facebook page about 10 Questions to ask when you read a story about science or medicine. I like what Steve Buist of the Hamilton Spectator has to say and to be sure, this is a good, simple guide that will get most people under way. However, I would argue there are four questions that are of greater importance in current science reporting:
- How many steps is this reporting away from the original science? A great deal of science “reporting” currently involves looking at a press release about a scientific study, or at times, a story about a press release about a study, or, as I have come to find out when trying to chase down a story’s origins, a story about a story about a press release about a study. Unfortunately, key details about the study are frequently omitted in the repetition of the information, rendering the science meaningless and misguiding. Equally when information is simplified or paraphrased for a new audience, nuance and important critical elements can be distorted to suggest things other than the point of the study.
- How many cherry picked results have been conflated into a single story? Reporters will often try to make a point about an idea, so they will look at several press releases about a single scientific topic, and choose results they believe will support a particular narrative. Of course, we all do this to a certain extent, with the exception that in science, you must not. You must take all of the results, and try to understand them in aggregate. Frequently the picture is complex and muddy, and this makes for a terrible story, but great science.
- Does the reader understand the topic in the study? Did the reporter understand the topic they were paraphrasing? It is easy to assume that just because we can parse all of the words in a sentence that we will necessarily understand what the author is trying to communicate. I can typically understand all of the individual words from a paper in an economics journal, but I would be hard pressed to explain the ideas to anyone. Some ideas, especially about cellular biology or neuroscience can be very complex, and although I am a neuroscientist, I can spend a lot of time trying to figure out exactly what a study showed, if anything. This can be particularly true if the technique or brain area or behaviour is unfamiliar to me.
- Do you have any clue what the statistics mean? Yes, more people in a study generally is better, but with a large enough group you can begin to find an effect for anything, and this is confusing unless you understand how hypothesis testing works, and what an effect size is. I am not saying you need to be statistical consultant, but in the words of Mark Twain, “There are three types of mendacity I can’t abide: Lies, damned lies, and statistics.” The bottom line is that statistics can be made to say many things and unless you are familiar with what test is allowed in a given circumstance, it can be hard to tell whether or not the result is as solid as it sounds.
Overall, if you are writing science stories, you are obliged to follow the advice of Einstein, who said “You do not really understand something unless you can explain it to your grandmother”. This involves the work of reading the ORIGINAL study, not the press release, and not someone’s paraphrase. It also involves reading any important supporting or contradictory background research. If you are unfamiliar with the techniques or ideas, it involves learning more about them. With respect to statistics, if you don’t understand them, in my view you are obliged at bare minimum to confess that lack of understanding.
The big point here is that with the “copy/paste” world we live in, we have an additional responsibility to be cautious when telling readers what to believe. Deliberate misinformation and misguided crusades in the name of weak or terrible science have been repeated to the point where the fiction is accepted as truth, resulting in misfortune and even death. The way of science is difficult, and fraught with enough peril and error— and at times, even deliberate deception. If you are going to report on science, at least do the hard work of thinking the issues through before you repeat them to someone else.