Flexible Dataset Distillation: Learn Labels Instead of Images

Abstract

We study the problem of dataset distillation - creating a small set of synthetic examples capable of training a good model. In particular, we study the problem of label distillation - creating synthetic labels for a small set of real images, and show it to be more effective than the prior image-based approach to dataset distillation. Methodologically, we introduce a more robust and flexible meta-learning algorithm for distillation, as well as an effective first-order strategy based on convex optimization layers. Distilling labels with our new algorithm leads to improved results over prior image-based distillation. More importantly, it leads to clear improvements in flexibility of the distilled dataset in terms of compatibility with off-the-shelf optimizers and diverse neural architectures. Interestingly, label distillation can also be applied across datasets, for example enabling learning Japanese character recognition by training only on synthetically labeled English letters.

arXiv commentPresented at the 4th Workshop on Meta-Learning (MetaLearn) at NeurIPS 2020

References in corpus (12)

• Distilling the Knowledge in a Neural Network
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Neural Architecture Search: A Survey
• The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
• On First-Order Meta-Learning Algorithms
• EMNIST: an extension of MNIST to handwritten letters
• Gradient-based Hyperparameter Optimization through Reversible Learning
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• Soft-Label Dataset Distillation and Text Dataset Distillation
• Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
• Dataset Distillation
• Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Estimators for Reinforcement Learning

Cited by in corpus (10)

• Knowledge Distillation: A Survey
• Dataset Condensation with Gradient Matching
• Dataset Condensation with Differentiable Siamese Augmentation
• Dataset Distillation with Infinitely Wide Convolutional Networks
• Graph Condensation for Graph Neural Networks
• Dataset Condensation with Distribution Matching
• Data Distillation for Text Classification
• SecDD: Efficient and Secure Method for Remotely Training Neural Networks
• Dataset Meta-Learning from Kernel Ridge-Regression
• EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization