Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflow
arXiv:2307.07367 · doi:10.1093/pnasnexus/pgae400
Abstract
Large language models like ChatGPT efficiently provide users with information about various topics, presenting a potential substitute for searching the web and asking people for help online. But since users interact privately with the model, these models may drastically reduce the amount of publicly available human-generated data and knowledge resources. This substitution can present a significant problem in securing training data for future models. In this work, we investigate how the release of ChatGPT changed human-generated open data on the web by analyzing the activity on Stack Overflow, the leading online Q\&A platform for computer programming. We find that relative to its Russian and Chinese counterparts, where access to ChatGPT is limited, and to similar forums for mathematics, where ChatGPT is less capable, activity on Stack Overflow significantly decreased. A difference-in-differences model estimates a 16\% decrease in weekly posts on Stack Overflow. This effect increases in magnitude over time, and is larger for posts related to the most widely used programming languages. Posts made after ChatGPT get similar voting scores than before, suggesting that ChatGPT is not merely displacing duplicate or low-quality content. These results suggest that more users are adopting large language models to answer questions and they are better substitutes for Stack Overflow for languages for which they have more training data. Using models like ChatGPT may be more efficient for solving certain programming problems, but its widespread adoption and the resulting shift away from public exchange on the web will limit the open data people and models can learn from in the future.
References in corpus (14)
- A Comprehensive Overview of Large Language Models
- The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
- OpenAssistant Conversations -- Democratizing Large Language Model Alignment
- What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks
- Will we run out of data? Limits of LLM scaling based on human-generated data
- Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks
- Astronomia ex machina: a history, primer, and outlook on neural networks in astronomy
- The False Promise of Imitating Proprietary LLMs
- Estimating Digital Product Trade through Corporate Revenue Data
- ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience
- The Unintended Consequences of Censoring Digital Technology -- Evidence from Italy's ChatGPT Ban
- Proving membership in LLM pretraining data via data watermarks
- Fairness Feedback Loops: Training on Synthetic Data Amplifies Bias
- Addressing Compiler Errors: Stack Overflow or Large Language Models?