Coresets via Bilevel Optimization for Continual Learning and Streaming

Abstract

Coresets are small data summaries that are sufficient for model training. They can be maintained online, enabling efficient handling of large data streams under resource constraints. However, existing constructions are limited to simple models such as k-means and logistic regression. In this work, we propose a novel coreset construction via cardinality-constrained bilevel optimization. We show how our framework can efficiently generate coresets for deep neural networks, and demonstrate its empirical benefits in continual learning and in streaming settings.

arXiv commentNeurIPS 2020

References in corpus (24)

• Overcoming catastrophic forgetting in neural networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• DARTS: Differentiable Architecture Search
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Understanding Black-box Predictions via Influence Functions
• Continual Learning Through Synaptic Intelligence
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Learning to Reweight Examples for Robust Deep Learning
• Deep Neural Networks as Gaussian Processes
• On Exact Computation with an Infinitely Wide Neural Net
• On Tiny Episodic Memories in Continual Learning
• BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning
• Submodular meets Spectral: Greedy Algorithms for Subset Selection, Sparse Approximation and Dictionary Selection
• Towards Robust Evaluations of Continual Learning
• Coresets for Scalable Bayesian Logistic Regression
• Bilevel Programming for Hyperparameter Optimization and Meta-Learning
• Guarantees for Greedy Maximization of Non-submodular Functions with Applications
• Selection via Proxy: Efficient Data Selection for Deep Learning
• Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
• Functional Regularisation for Continual Learning with Gaussian Processes
• Automated Scalable Bayesian Inference via Hilbert Coresets
• Approximation Methods for Bilevel Programming
• Submodular Maximization Beyond Non-negativity: Guarantees, Fast Algorithms, and Applications

Cited by in corpus (25)

• Data-Centric Green AI: An Exploratory Empirical Study
• Training Data Influence Analysis and Estimation: A Survey
• Recent Advances of Continual Learning in Computer Vision: An Overview
• Online Coreset Selection for Rehearsal-based Continual Learning
• Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guarantees
• Dataset Distillation with Infinitely Wide Convolutional Networks
• Cost-effective On-device Continual Learning over Memory Hierarchy with Miro
• The Effectiveness of Memory Replay in Large Scale Continual Learning
• Rainbow Memory: Continual Learning with a Memory of Diverse Samples
• Self-supervised On-device Federated Learning from Unlabeled Streams
• Carousel Memory: Rethinking the Design of Episodic Memory for Continual Learning
• A First Look at Class Incremental Learning in Deep Learning Mobile Traffic Classification
• Dataset Condensation with Distribution Matching
• DaCapo: Accelerating Continuous Learning in Autonomous Systems for Video Analytics
• FairIF: Boosting Fairness in Deep Learning via Influence Functions with Validation Set Sensitive Attributes
• Dataset Meta-Learning from Kernel Ridge-Regression
• Posterior Meta-Replay for Continual Learning
• Enabling On-Device Self-Supervised Contrastive Learning With Selective Data Contrast
• Improved Regret Bound and Experience Replay in Regularized Policy Iteration
• Bilevel Continual Learning
• One Backward from Ten Forward, Subsampling for Large-Scale Deep Learning
• Online Continual Learning in Image Classification: An Empirical Survey
• Condensed Composite Memory Continual Learning
• A resource-efficient method for repeated HPO and NAS problems
• Sum of Ranked Range Loss for Supervised Learning