Data Summarization via Bilevel Optimization

Abstract

The increasing availability of massive data sets poses a series of challenges for machine learning. Prominent among these is the need to learn models under hardware or human resource constraints. In such resource-constrained settings, a simple yet powerful approach is to operate on small subsets of the data. Coresets are weighted subsets of the data that provide approximation guarantees for the optimization objective. However, existing coreset constructions are highly model-specific and are limited to simple models such as linear regression, logistic regression, and $k$-means. In this work, we propose a generic coreset construction framework that formulates the coreset selection as a cardinality-constrained bilevel optimization problem. In contrast to existing approaches, our framework does not require model-specific adaptations and applies to any twice differentiable model, including neural networks. We show the effectiveness of our framework for a wide range of models in various settings, including training non-convex models online and batch active learning.

References in corpus (18)

• Overcoming catastrophic forgetting in neural networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Wide Residual Networks
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• DARTS: Differentiable Architecture Search
• Understanding Black-box Predictions via Influence Functions
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Continual Learning Through Synaptic Intelligence
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Compressed Sensing using Generative Models
• An Empirical Study of Example Forgetting during Deep Neural Network Learning
• Selection via Proxy: Efficient Data Selection for Deep Learning
• Automated Scalable Bayesian Inference via Hilbert Coresets
• Coresets for Data-efficient Training of Machine Learning Models
• GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning
• Combining MixMatch and Active Learning for Better Accuracy with Fewer Labels
• Consistency-based Semi-supervised Active Learning: Towards Minimizing Labeling Cost
• Instance-Optimal Compressed Sensing via Posterior Sampling