GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training
arXiv:2103.00123
Abstract
The great success of modern machine learning models on large datasets is contingent on extensive computational resources with high financial and environmental costs. One way to address this is by extracting subsets that generalize on par with the full data. In this work, we propose a general framework, GRAD-MATCH, which finds subsets that closely match the gradient of the training or validation set. We find such subsets effectively using an orthogonal matching pursuit algorithm. We show rigorous theoretical and convergence guarantees of the proposed algorithm and, through our extensive experiments on real-world datasets, show the effectiveness of our proposed framework. We show that GRAD-MATCH significantly and consistently outperforms several recent data-selection algorithms and achieves the best accuracy-efficiency trade-off. GRAD-MATCH is available as a part of the CORDS toolkit: \url{https://github.com/decile-team/cords}.
arXiv commentTo appear in Proceedings of the 38 th International Conference on Machine Learning, PMLR 139, 2021
References in corpus (9)
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
- Submodular meets Spectral: Greedy Algorithms for Subset Selection, Sparse Approximation and Dictionary Selection
- The Cost of Training NLP Models: A Concise Overview
- An Empirical Study of Example Forgetting during Deep Neural Network Learning
- Coresets for Data-efficient Training of Machine Learning Models
- GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning
- Coresets for Robust Training of Neural Networks against Noisy Labels
- Non-submodular Function Maximization subject to a Matroid Constraint, with Applications