Caveat Emptor, Computational Social Science: Large-Scale Missing Data in a Widely-Published Reddit Corpus

Abstract

As researchers use computational methods to study complex social behaviors at scale, the validity of this computational social science depends on the integrity of the data. On July 2, 2015, Jason Baumgartner published a dataset advertised to include ``every publicly available Reddit comment'' which was quickly shared on Bittorrent and the Internet Archive. This data quickly became the basis of many academic papers on topics including machine learning, social behavior, politics, breaking news, and hate speech. We have discovered substantial gaps and limitations in this dataset which may contribute to bias in the findings of that research. In this paper, we document the dataset, substantial missing observations in the dataset, and the risks to research validity from those gaps. In summary, we identify strong risks to research that considers user histories or network analysis, moderate risks to research that compares counts of participation, and lesser risk to machine learning research that avoids making representative claims about behavior and participation on Reddit.

References in corpus (9)

• Twitter mood predicts the stock market
• Circadian pattern and burstiness in mobile phone communication
• A Web of Hate: Tackling Hateful Speech in Online Social Spaces
• All Who Wander: On the Prevalence and Characteristics of Multi-community Engagement
• Averaging Gone Wrong: Using Time-Aware Analyses to Better Understand Behavior
• Cats and Captions vs. Creators and the Clock: Comparing Multimodal Content to Context in Predicting Relative Popularity
• Science, AskScience, and BadScience: On the Coexistence of Highly Related Communities
• What do Vegans do in their Spare Time? Latent Interest Detection in Multi-Community Networks
• Time Is of the Essence: Analyzing the Effect of Vertex-Joining Time on Complex Network Evolution

Cited by in corpus (16)

• The anatomy of Reddit: An overview of academic research
• Patterns of Routes of Administration and Drug Tampering for Nonmedical Opioid Consumption: Data Mining and Content Analysis of Reddit Discussions
• Firsthand Opiates Abuse on Social Media: Monitoring Geospatial Patterns of Interest Through a Digital Cohort
• Ten Social Dimensions of Conversations and Relationships
• Are All Successful Communities Alike? Characterizing and Predicting the Success of Online Communities
• Incorrect Data in the Widely Used Inside Airbnb Dataset
• "This is why we play": Characterizing Online Fan Communities of the NBA Teams
• SemAxis: A Lightweight Framework to Characterize Domain-Specific Word Semantics Beyond Sentiment
• Tracing Community Genealogy: How New Communities Emerge from the Old
• The Impact of Data Persistence Bias on Social Media Studies
• Estimating Community Feedback Effect on Topic Choice in Social Media with Predictive Modeling
• Quantifying Voter Biases in Online Platforms: An Instrumental Variable Approach
• Variation across Scales: Measurement Fidelity under Twitter Data Sampling
• Learning Invariant Representations of Social Media Users
• Assembly in populations of social networks
• Characterizing behavioral trends in a community driven discussion platform