Dataset Condensation with Differentiable Siamese Augmentation
arXiv:2102.08259
Abstract
In many machine learning problems, large-scale datasets have become the de-facto standard to train state-of-the-art deep networks at the price of heavy computation load. In this paper, we focus on condensing large training sets into significantly smaller synthetic sets which can be used to train deep neural networks from scratch with minimum drop in performance. Inspired from the recent training set synthesis methods, we propose Differentiable Siamese Augmentation that enables effective use of data augmentation to synthesize more informative synthetic images and thus achieves better performance when training networks with augmentations. Experiments on multiple image classification benchmarks demonstrate that the proposed method obtains substantial gains over the state-of-the-art, 7% improvements on CIFAR10 and CIFAR100 datasets. We show with only less than 1% data that our method achieves 99.6%, 94.9%, 88.5%, 71.5% relative performance on MNIST, FashionMNIST, SVHN, CIFAR10 respectively. We also explore the use of our method in continual learning and neural architecture search, and show promising results.
References in corpus (19)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Distilling the Knowledge in a Neural Network
- Conditional Generative Adversarial Nets
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Improved Regularization of Convolutional Neural Networks with Cutout
- Neural Architecture Search: A Survey
- VQA: Visual Question Answering
- YouTube-8M: A Large-Scale Video Classification Benchmark
- AutoAugment: Learning Augmentation Policies from Data
- Exploring Simple Siamese Representation Learning
- Image Augmentations for GAN Training
- An Empirical Study of Example Forgetting during Deep Neural Network Learning
- Data augmentation instead of explicit regularization
- Soft-Label Dataset Distillation and Text Dataset Distillation
- Flexible Dataset Distillation: Learn Labels Instead of Images
- Using Small Proxy Datasets to Accelerate Hyperparameter Search
- Dataset Meta-Learning from Kernel Ridge-Regression