Soft-Label Dataset Distillation and Text Dataset Distillation

Abstract

Dataset distillation is a method for reducing dataset sizes by learning a small number of synthetic samples containing all the information of a large dataset. This has several benefits like speeding up model training, reducing energy consumption, and reducing required storage space. Currently, each synthetic sample is assigned a single `hard' label, and also, dataset distillation can currently only be used with image data. We propose to simultaneously distill both images and their labels, thus assigning each synthetic sample a `soft' label (a distribution of labels). Our algorithm increases accuracy by 2-4% over the original algorithm for several image classification tasks. Using `soft' labels also enables distilled datasets to consist of fewer samples than there are classes as each sample can encode information for multiple classes. For example, training a LeNet model with 10 distilled images (one per class) results in over 96% accuracy on MNIST, and almost 92% accuracy when trained on just 5 distilled images. We also extend the dataset distillation algorithm to distill sequential datasets including texts. We demonstrate that text distillation outperforms other methods across multiple datasets. For example, models attain almost their original accuracy on the IMDB sentiment analysis task using just 20 distilled sentences. Our code can be found at $\text{href}$.

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Federated Learning: Challenges, Methods, and Future Directions
• Generative Adversarial Text to Image Synthesis
• End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
• Sequence-Level Knowledge Distillation
• Measuring the Intrinsic Dimension of Objective Landscapes
• Soft-Label Anonymous Gastric X-ray Image Distillation

Cited by in corpus (14)

• A Comprehensive Survey of Dataset Distillation
• Dataset Condensation with Gradient Matching
• Distilled One-Shot Federated Learning
• Dataset Condensation with Differentiable Siamese Augmentation
• Federated Learning Meets Natural Language Processing: A Survey
• Compressed Gastric Image Generation Based on Soft-Label Dataset Distillation for Medical Data Sharing
• Federated Learning via Synthetic Data
• Dataset Distillation with Infinitely Wide Convolutional Networks
• Flexible Dataset Distillation: Learn Labels Instead of Images
• Dataset Condensation with Distribution Matching
• SecDD: Efficient and Secure Method for Remotely Training Neural Networks
• Dataset Meta-Learning from Kernel Ridge-Regression
• One Line To Rule Them All: Generating LO-Shot Soft-Label Prototypes
• Bridge Networks: Relating Inputs through Vector-Symbolic Manipulations