Reproducibility may be the key idea students need to balance trust in evidence with healthy skepticism
- Written by Sarah R. Supp, Associate Professor of Data Analytics, Denison University
Many people have been there.
The dinner party is going well until someone decides to introduce a controversial topic. In today’s world, that could be anything from vaccines to government budget cuts to immigration policy. Conversation starts to get heated. Finally, someone announces with great authority that a scientific study supports their position. This causes the discussion to come to an abrupt halt because the dinner guests disagree on their belief in scientific evidence. Some may believe science always speaks the truth, some may think science can never be trusted, and others may disagree on which studies with contradicting claims are “right.”
How can the dinner party – or society – move beyond this kind of impasse? In today’s world of misinformation and disinformation[1], healthy skepticism is essential[2]. At the same time, much scientific work is rigorous and trustworthy. How do you reach a healthy balance between trust and skepticism? How can researchers increase the transparency of their work to make it possible to evaluate how much confidence the public should have in any particular study?
As teachers and scholars[3], we see these problems in our own classrooms and in our students – and they are mirrored in society.
The concept of reproducibility may offer important answers to these questions.
Reproducibility is what it sounds like: reproducing results. In some ways, reproducibility is like a well-written recipe, such as a recipe for an award-winning cake at the county fair. To help others reproduce their cake, the proud prizewinner must clearly document the ingredients used and then describe each step of the process by which the ingredients were transformed into a cake. If others can follow the directions and come up with a cake of the same quality, then the recipe is reproducible.
Think of the English scholar who claims that Shakespeare did not author a play that has historically been attributed to him. A critical reader will want to know exactly how they arrived at that conclusion. What is the evidence? How was it chosen and interpreted? By parsing the analysis step by step, reproducibility allows a critical reader to gauge the strength of any kind of argument.
We are a group of researchers and professors[4] from a wide range of disciplines who came together to discuss how we use reproducibility in our teaching and research.
Based on our expertise and the students we encounter, we collectively see a need for higher-education students to learn about reproducibility in their classes, across all majors. It has the potential to benefit students and, ultimately, to enhance the quality of public discourse.
The foundation of credibility
Reproducibility[5] has always been a foundation of good science because it allows researchers to scrutinize each other’s studies for rigor and credibility and expand upon prior work to make new discoveries. Researchers are increasingly paying attention to reproducibility in the natural sciences, such as physics and medicine, and in the social sciences, such as economics and environmental studies. Even researchers in the humanities, such as history and philosophy, are concerned with reproducibility in studies involving analysis of texts and evidence[6], especially with digital and computational methods[7]. Increased interest in transparency and accessibility has followed the rising importance of computer algorithms and numerical analysis in research. This work should be reproducible, but it often remains opaque.
Broadly, research is reproducible if it answers the question: “How do you know?” − such that another researcher could theoretically repeat the study and produce consistent results.
Reproducible research is explicit about the materials and methods that were used in a study to make discoveries and come to conclusions. Materials include everything from scientific instruments such as a tensiometer[8] measuring soil moisture to surveys asking people about their daily diet. They also include digital data such as spreadsheets, digitized historic texts, satellite images and more. Methods include how researchers make observations and analyze data.
To reproduce a social science study, for example, we would ask: What is the central question or hypothesis? Who was in the study? How many individuals were included? What were they asked? After data was collected, how was it cleaned and prepared for analysis? How exactly was the analysis run?
Proper documentation of all these steps, plus making available the original data from the study, allows other scientists to redo the research, evaluate the decisions made during the process of gathering and analyzing information, and assess the credibility of the findings.
This short video, made by the National Academies, explains the key concepts in reproducing scientific findings and notes ways the process can be improved.Over the past 20 years, the need for reproducibility has become increasingly important. Scientists have discovered that some published studies are too poorly documented[9] for others to repeat, lack verified data sources[10], are questionably[11] designed[12], or even fraudulent[13].
Putting reproducibility to work: An example
A highly contentious, retracted study[14] from 1998 linked the measles, mumps and rubella (MMR) vaccine and autism. Scientists and journalists used their understanding of reproducibility to discover the flaws in the study.
The central question of the study was not about vaccines but aimed to explore a possible relationship between colitis − an inflammation of the large intestine − and developmental disorders. The authors explicitly wrote, “We did not prove an association between measles, mumps, and rubella vaccine and the syndrome described.”
The study observed just 12 patients who were referred to the authors’ gastroenterology clinic and had histories of recent behavioral disorders, including autism. This sample of children is simply too small and selective to be able to make definitive conclusions.
In this study, the researchers translated children’s medical charts into summary tables for comparison. When a journalist attempted to reproduce the published data tables from the children’s medical histories, they found pervasive inconsistencies[15].
Reproducibility allows for corrections in research. The article was published in a respected journal, but it lacked transparency with regard to patient recruitment, data analysis and conflicts of interest. Whereas traditional peer review involves critical evaluation of a manuscript, reproducibility also opens the door to evaluating the underlying data and methods. When independent researchers attempted to reproduce this study, they found deep flaws. The article was retracted by the journal[16] and by most of its authors[17]. Independent research teams conducted more robust studies, finding no relationship between vaccines and autism[18].
Each research discipline has its own set of best practices for achieving reproducibility. Disciplines in which researchers use computational or statistical analysis require sharing the data and software code[19] for reproducing studies. In other disciplines, researchers interpret nonnumerical qualities of data sources such as interviews, historical texts, social media content and more. These disciplines are working to develop standards for sharing their data[20] and research designs[21] for reproducibility. Across disciplines, the core principles are the same: transparency of the evidence and arguments by which researchers arrived at their conclusions.
Reproducibility in the classroom
Colleges and universities are uniquely situated to promote reproducibility in research and public conversations. Critical thinking, effective communication and intellectual integrity, staples of higher-education mission statements, are all served by reproducibility.
Teaching faculty at colleges and universities have started taking some important steps toward incorporating reproducibility into a wide range of undergraduate and graduate courses. These include assignments to replicate[22] existing studies, training[23] in reproducible methods[24] to conduct and document original research, preregistration[25] of hypotheses and analysis plans[26], and tools to facilitate open collaboration among peers. A number of initiatives[27] to develop[28] and disseminate[29] resources for teaching reproducibility have been launched.
Despite some progress, reproducibility still needs a central place in higher education. It can be integrated into any course in which students weigh evidence, read published literature to make claims, or learn to conduct their own research. This change is urgently needed to train the next generation of researchers[30], but that is not the only reason.
Reproducibility is fundamental to constructing and communicating claims based on evidence. Through a reproducibility lens, students evaluate claims in published studies as contingent on the transparency and soundness of the evidence and analysis on which the claims are based. When faculty teach reproducibility as a core expectation from the beginning of a curriculum, they encourage students to internalize its principles in how they conduct their own research and engage with the research published by others.
Institutions of higher education already prioritize cultivating engaged, literate and critical citizens capable of solving the world’s most challenging contemporary problems. Teaching reproducibility equips students, and members of the public, with the skills they need to critically analyze claims in published research, in the media and even at dinner parties.
Also contributing to this article are participants in the 2024 Reproducibility and Replicability in the Liberal Arts workshop[31], funded by the Alliance to Advance Liberal Arts Colleges (AALAC) [in alphabetical order]: Ben Gebre-Medhin (Department of Sociology and Anthropology, Mount Holyoke College), Xavier Haro-Carrión (Department of Geography, Macalester College), Emmanuel Kaparakis (Quantitative Analysis Center, Wesleyan University), Scott LaCombe (Statistical and Data Sciences, Smith College), Matthew Lavin (Data Analytics Program, Denison University), Joseph J. Merry (Sociology Department, Furman University), Laurie Tupper (Department of Mathematics and Statistics, Mount Holyoke College).
References
- ^ misinformation and disinformation (doi.org)
- ^ skepticism is essential (www.kff.org)
- ^ teachers and scholars (doi.org)
- ^ group of researchers and professors (doi.org)
- ^ Reproducibility (book.the-turing-way.org)
- ^ analysis of texts and evidence (doi.org)
- ^ digital and computational methods (link.springer.com)
- ^ tensiometer (www.merriam-webster.com)
- ^ too poorly documented (doi.org)
- ^ lack verified data sources (www.science.org)
- ^ questionably (doi.org)
- ^ designed (doi.org)
- ^ fraudulent (doi.org)
- ^ highly contentious, retracted study (doi.org)
- ^ pervasive inconsistencies (www.bmj.com)
- ^ journal (www.thelancet.com)
- ^ most of its authors (www.thelancet.com)
- ^ no relationship between vaccines and autism (doi.org)
- ^ sharing the data and software code (doi.org)
- ^ data (qdr.syr.edu)
- ^ research designs (doi.org)
- ^ replicate (doi.org)
- ^ training (doi.org)
- ^ reproducible methods (www.projecttier.org)
- ^ preregistration (doi.org)
- ^ of hypotheses and analysis plans (doi.org)
- ^ initiatives (www.bitss.org)
- ^ develop (forrt.org)
- ^ disseminate (ciesin-geospatial.github.io)
- ^ train the next generation of researchers (doi.org)
- ^ Reproducibility and Replicability in the Liberal Arts workshop (doi.org)
Authors: Sarah R. Supp, Associate Professor of Data Analytics, Denison University
