Agenda

• Evaluation
• Social science’s replication crisis
• The philosophy of Open Science
• Exercise: Setting up GitHub

Evaluation

• Click this link

Or follow the QR code:

Replication Crisis

• A systemic issue in science where many studies fail to produce the same results when repeated
• Affects psychology, medicine, economics, biology, and other disciplines

Emergence of the Crisis

• Why Most Published Research Findings Are False (Ioannidis 2005):
  • “Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true.”
  • Also see: Why most psychological research findings are not even wrong (Scheel: 2022)
• Estimating the reproducibility of psychological science (Open Science Collaboration 2015) – Only 39% of psychology experiments replicated

Causes of the Replication Crisis

• Publication Bias: Preference for novel, positive results
• File drawer problem: Null results more likely go unpublished
• P-Hacking: Manipulating data until statistically significant results appear (e.g., \(p<0.05\))
• Low statistical power: Small sample sizes lead to unreliable results
• Lack of transparency: Inadequate sharing of data, methods, and materials

Impact of the Crisis

• Loss of trust in published findings
• Waste of resources on irreproducible studies
• Misguided policies or treatments based on flawed research
• Erosion of public trust in science

Replication Studies in the Social Sciences

• Studies aimed at repeating previous studies to verify their results
• (Usually) same methods, conditions, and context as the original study
• Idea: Do teams of researchers come to the same conclusion testing the same hypothesis? (“many-analysts approach”)

Replication Studies: Examples

• Pioneering: Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results (Silberzahn et al. 2018); but see reply by Auspurg & Brüderl (2021)
• Mass Reproducibility and Replicability: A New Hope (Brodeur et al. 2024), also see Institute For Replication
• A many-analysts approach to the relation between religiosity and well-being (Hoogeveen et al. 2022)
• Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty (Breznau et al. 2022)

Criticism of Replication Studies

• Toward a more credible assessment of the credibility of science by many-analyst studies (Auspurg & Brüderl 2024)
  • “existing many-analyst studies have likely provided a too pessimistic picture of the health of science. Overestimation of uncertainty in scientific results is likely to threaten public confidence in science. Other scientists, the public, or politicians may think we are just “poking around in the fog”“
  • “To make many-analyst studies even more credible, they should share all data and syntax code used by the teams and PIs and also provide raw data on analysts’ estimates before PIs applied rescaling transformations.”
• But: The replication crisis has led to positive structural, procedural, and community changes (Korbmacher et al. 2023)

Addressing the Crisis

• Pre-Registration: Register hypotheses, methods, and analyses before conducting research
• Open Science: Sharing datasets, code, and protocols
• Replication Studies
• Statistical Reforms: Bayesian methods, confidence intervals, larger sample sizes
• Cultural change in science: Shift focus from novelty to reliability (e. g.: PLOS ONE: “The editors make decisions on submissions based on scientific rigor, regardless of novelty.”)

Introduction to Open Science

• A movement to make scientific research accessible, transparent, and reproducible
• Core principles
  • Open data
  • Open methods
  • Open access
  • Open collaboration

The Core Tenets of Open Science

• Transparency
  • Sharing data, methods, and code
  • Clear documentation of workflows
• Reproducibility
  • Ensuring results can be independently verified
  • Avoiding the replication crisis
• Collaboration
  • Facilitating interdisciplinary and global teamwork
  • Examples: Open-source projects, collaborative platforms

Tools for Open Science

• Code Sharing
  • Platforms: OSF, GitHub
  • Tools: Jupyter Notebooks, RMarkdown
• Data Sharing
  • Zenodo, Harvard Dataverse, Gesis Data Services
• Open Access
  • Preprint Servers: OSF, arXiv, SSRN
  • Journals: E. g. PLOS ONE, Sociological Science
  • Repositories: E. g. Social Science Open Access Repository

A (Slowly) Changing Culture?

• (Some) journals are adopting open data policies
  • Political Science Research and Methods
  • European Sociological Review
  • European Societies