Letter Review:
Science Fictions: How FRAUD, BIAS, NEGLIGENCE, and HYPE Undermine the Search For Truth. Stuart Ritchie. Metropolitan Books. 2020.
353 p. Includes epilogue, appendix, extensive citations, and footnotes. Hard cover; sewn and glued. 1.3 pounds. First discovered at Eureka Springs Carnegie Library; purchased at Alibris.com.
Dear Mr. Ritchie,
I enjoyed your book Science Fictions. While it was written mostly for the science community, you also made the book accessible to the lay community, so I am moved by your own passion for this topic.
As the subtitle suggests, you see four main categories of mistakes made by scientists and the modern scientific culture that ultimately undermine the integrity of science. Of the four, it is my impression that you regard Bias as a huge component of what ails science today, especially regarding p-values. And in the end, you give a nice summary of how to read science papers, which is the topic I wish to engage with you later.
First, you set the stage with some background terminology; I want to look at some of it briefly to ensure I’m on the same page with you.
I appreciate how you succinctly explained Replication and Reproducibility, both terms I have seen often but never compared before. Replication: ask the same question with a different set of data; does one get the same/similar answer or not? Reproducibility: have a second person analyze the same data as the researcher; does the second person get the same results?
In your opening pitch about Fraud, you mention how science relies on trust of both individuals and institutions; but especially how the culture of science so implicitly relies on trust/honesty that it can become blinding to instances of fraud. You mention some infamous examples of the science researcher who committed fraud in cloning, and a medical scientist who was quite fraudulent with his anesthesiology research.
In fact, I appreciate the anecdotal stories you pepper throughout the book to illustrate your topics. Many, if not most, were recognizable from news I’ve read, heard or seen over the years. Seeing them put into the context of ‘science fictions’ helps put all the pieces together.
You mention the effects of fraud on scientists and the scientific culture, and my first thoughts as I began reading them were that these effects are common across any community of people (waste of time, waste of money, demoralizing, polluting); until, that is, we get to the last bad effect: misleading medical treatment suggestions to patients. That’s sobering: to think a doctor could be sincerely prescribing something that they have been misled into believing to be effective due to fraudulent research.
In talking about Negligence, you mention in some detail about the importance of properly sized samples in order to give the study proper statistical power. What’s fascinating to me about this chapter is the chilling comment from you that studies with a too-low sample size just shouldn’t have been done at all. This seems especially morally true regarding any studies that involve people or animals, and especially studies that require animal deaths. I agree wholeheartedly.
Your chapter on Hype is one of my favorites; I see hype all the time in my field of work in the grocery business. There are so many bogus and hyped-up claims about what a food can do for you, that it can create a backlash against science when people see competing and sometimes contradictory claims.
Back to your second concern, Bias, in which you talk extensively about p-values. These p-values are subsequently mentioned in the following chapters and figure large in your overall indictment of science’s current problems. I have seen p-values mentioned in magazine and in science news articles, so I am now doubly motivated to understand this concept of p-value.
Here’s how I understand it: p-value is a statistical number, and the smaller that number the better for the study’s conclusions. Much of science uses the arbitrary cut-off of .05, which seems to be influential in the large number of studies with outcomes hovering right around that magical .05. But other science studies might need a different benchmark; you mention the astounding example of the Hadron Collider experiments needing a p-value of .000005.
The p-value was originally developed to avoid valuing random chance. It was designed as a statistical approach to ensure data was evaluated appropriately and that random ‘noise’ found in any data set of numbers is excluded.
P-hacking is when a scientist or statistician repeatedly crunches the same set of numbers until they find some p-value below .05. That p-value then becomes the center-piece of the study, either ostensibly supporting the original hypotheses or informing a new hypothesis. P-hacking overfits the data; it finds a pattern in the noise of the data, but this pattern is not generalizable to a broader hypothesis. Thus, there will be a problem with replicating the study, or even in reproducing the results.
P-values thus become a central part of your argument, for p-values are critical to a study being published, and subsequent funding for that scientist depends on publishing. The fixation on finding an appropriate p-value to buttress a study totally circumvents the original purpose of calculating it, which was to avoid believing in the random noise of the data. This fixation is a large contributing factor to fraud, negligence, and hype as well as bias.
I hope I got that description of the importance of p-values correct, for I still could not explain to another person what exactly a p-value is: what the ‘p’ stands for and what numbers or formulas are used to calculate it. What I do take away from your book is to be cautious when a study or science-oriented book is based on a whole lot of p-values hovering right around .05.
My question for you comes from the book’s ending, in which you list 10 things to look for when reading a Science Paper. While this list is informative for me, I have to admit I don’t read many original source materials; I more often read books for lay-readers about science or based on science research. Because of that, and I suspect many lay readers might agree, I wish you had included information on how to read such science-based books.
Here is your list, with my elaborations on reading books:
- “Who is the author, what university or lab are they associated with, and what background do they have?” This is appropriate to non-scientific books as well.
- “How transparent is the paper. Was the study pre-registered and is the data available?” For books, do authors reveal their own biases and any other relevant information a reader should know to help them judge the author’s (or the book’s) veracity?
- “How well-designed is the study; does it include Blinding, Randomization, Control Groups?” Does the book cite such high-quality studies; this question is relevant to the next several points.
- “How appropriate is the sample size, how many of the original study participants were left out in the final analysis, and why?”
- “How big is the effect? Are there lots of p-values hovering right around the magic point of .05?”
- “Are the study’s inferences appropriate and not confusing correlation with causation?” One must beware of this to a significant degree, especially in books by medical doctors, evolution psychology topics, and writers who like to jump to conclusions.
- “What biases are present, and how much are they acknowledged?” I conflated this one with point #2.
- “How plausible is the study?” You ask us to imagine being a part of the study: would we find the questions asked, the routines we had to follow, the instructions given, to be plausible approaches to answering the study’s question(s)?
- “Has the study been replicated?” When reading books, again, are the cited studies ones that have been replicated. Is the author predominantly using studies that are well-designed, with appropriate sample sizes, that show statistically truly significant results, with appropriate conclusions to the studies’ designs?
- “What do other scientists think?” When I saw this, I immediately thought of the big news of homo florensis, or The Hobbit of Indonesia. I waited after hearing the first news reports, and then in the months to years afterwards, I read assiduously anything in Scientific American, Nature, and Science that might mention this topic; always I was searching for what other scientists thought of this discovery. But how do we apply this when reading books?
Could you elaborate even further on how lay-readers can approach books by and about science—or claiming to be based on science—with a critical eye looking out for fraud, bias, negligence, and hype? There will always be low-hanging fruit: books that are just so obviously sensationalistic, even when purporting to be based on science (often diet or self-help books). I usually avoid such books; I’m more interested in how to read books with a critical eye that have been written and promoted with the claim to objectivity, but still have a clear and strong concluding opinion. I wonder, for example, about many NY Times’ Best Sellers on Physics, Biology, or Evolution. How can I read them without getting caught up in their ‘science fictions’?
Thank you.
Other Reviews that mention this post: Simple Steps to Foot Pain Relief, How To Read A Book, Essay on The Science of Storytelling