Dataset Condensation with Gradient Matching

Abstract

As the state-of-the-art machine learning methods in many fields rely on larger datasets, storing datasets and training models on them become significantly more expensive. This paper proposes a training set synthesis technique for data-efficient learning, called Dataset Condensation, that learns to condense large dataset into a small set of informative synthetic samples for training deep neural networks from scratch. We formulate this goal as a gradient matching problem between the gradients of deep neural network weights that are trained on the original and our synthetic data. We rigorously evaluate its performance in several computer vision benchmarks and demonstrate that it significantly outperforms the state-of-the-art methods. Finally we explore the use of our method in continual learning and neural architecture search and report promising gains when limited memory and computations are available.

References in corpus (20)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Practical Bayesian Optimization of Machine Learning Algorithms
• Prototypical Networks for Few-shot Learning
• Neural Architecture Search with Reinforcement Learning
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• FitNets: Hints for Thin Deep Nets
• Do Deep Nets Really Need to be Deep?
• Understanding Black-box Predictions via Influence Functions
• iDLG: Improved Deep Leakage from Gradients
• Data-Free Knowledge Distillation for Deep Neural Networks
• Soft-Label Dataset Distillation and Text Dataset Distillation
• Zero-Shot Knowledge Distillation in Deep Networks
• Distilled One-Shot Federated Learning
• Federated Learning via Synthetic Data
• Flexible Dataset Distillation: Learn Labels Instead of Images

Cited by in corpus (7)

• Graph Condensation for Graph Neural Networks
• Less is More: Proxy Datasets in NAS approaches
• Interpretable Data Fusion for Distributed Learning: A Representative Approach via Gradient Matching
• Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratch
• A Continual and Incremental Learning Approach for TinyML On-device Training Using Dataset Distillation and Model Size Adaption
• SecDD: Efficient and Secure Method for Remotely Training Neural Networks
• Condensed Composite Memory Continual Learning