Data Shapley: Equitable Valuation of Data for Machine Learning

Abstract

As data becomes the fuel driving technological and economic growth, a fundamental challenge is how to quantify the value of data in algorithmic predictions and decisions. For example, in healthcare and consumer markets, it has been suggested that individuals should be compensated for the data that they generate, but it is not clear what is an equitable valuation for individual data. In this work, we develop a principled framework to address data valuation in the context of supervised machine learning. Given a learning algorithm trained on $n$ data points to produce a predictor, we propose data Shapley as a metric to quantify the value of each training datum to the predictor performance. Data Shapley value uniquely satisfies several natural properties of equitable data valuation. We develop Monte Carlo and gradient-based methods to efficiently estimate data Shapley values in practical settings where complex learning algorithms, including neural networks, are trained on large datasets. In addition to being equitable, extensive experiments across biomedical, image and synthetic data demonstrate that data Shapley has several other benefits: 1) it is more powerful than the popular leave-one-out or leverage score in providing insight on what data is more valuable for a given learning task; 2) low Shapley value data effectively capture outliers and corruptions; 3) high Shapley value data inform what type of new data to acquire to improve the predictor.

Cited by in corpus (42)

• On the Opportunities and Risks of Foundation Models
• Training Data Influence Analysis and Estimation: A Survey
• Neuron Shapley: Discovering the Responsible Neurons
• Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning
• Data Valuation using Reinforcement Learning
• Hierarchically Fair Federated Learning
• Approximate Data Deletion from Machine Learning Models
• A Distributional Framework for Data Valuation
• HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networks
• Causative Cyberattacks on Online Learning-based Automated Demand Response Systems
• Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learning
• Efficient computation and analysis of distributional Shapley values
• Data Market Platforms: Trading Data Assets to Solve Data Problems
• Online Class-Incremental Continual Learning with Adversarial Shapley Value
• FedCCEA : A Practical Approach of Client Contribution Evaluation for Federated Learning
• Understanding Instance-based Interpretability of Variational Auto-Encoders
• RelatIF: Identifying Explanatory Training Examples via Relative Influence
• Who's responsible? Jointly quantifying the contribution of the learning algorithm and training data
• Learning Fast Sample Re-weighting Without Reward Data
• Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation
• Representation Matters: Assessing the Importance of Subgroup Allocations in Training Data
• Dealer: End-to-End Data Marketplace with Model-based Pricing
• DIVINE: Diverse Influential Training Points for Data Visualization and Model Refinement
• Absolute Shapley Value
• Faithful Edge Federated Learning: Scalability and Privacy
• A Unified Framework for Task-Driven Data Quality Management
• Explaining medical AI performance disparities across sites with confounder Shapley value analysis
• One for One, or All for All: Equilibria and Optimality of Collaboration in Federated Learning
• Mapping the Potential and Pitfalls of "Data Dividends" as a Means of Sharing the Profits of Artificial Intelligence
• Data Sharing Markets
• Estimating informativeness of samples with Smooth Unique Information
• Energy-Based Learning for Cooperative Games, with Applications to Valuation Problems in Machine Learning
• Data Appraisal Without Data Sharing
• Interpretable feature subset selection: A Shapley value based approach
• Evaluating the Values of Sources in Transfer Learning
• Efficient Client Contribution Evaluation for Horizontal Federated Learning
• ModelPred: A Framework for Predicting Trained Model from Training Data
• Rigorous Explanation of Inference on Probabilistic Graphical Models
• Data-Driven Methods for Balancing Fairness and Efficiency in Ride-Pooling
• HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations
• Model-Agnostic Explanations using Minimal Forcing Subsets
• Learning to Transfer Learn: Reinforcement Learning-Based Selection for Adaptive Transfer Learning