A Comprehensive Survey of Dataset Distillation
arXiv:2301.05603 · doi:10.1109/TPAMI.2023.3322540
Abstract
Deep learning technology has developed unprecedentedly in the last decade and has become the primary choice in many application domains. This progress is mainly attributed to a systematic collaboration in which rapidly growing computing resources encourage advanced algorithms to deal with massive data. However, it has gradually become challenging to handle the unlimited growth of data with limited computing power. To this end, diverse approaches are proposed to improve data processing efficiency. Dataset distillation, a dataset reduction method, addresses this problem by synthesizing a small typical dataset from substantial data and has attracted much attention from the deep learning community. Existing dataset distillation methods can be taxonomized into meta-learning and data matching frameworks according to whether they explicitly mimic the performance of target data. Although dataset distillation has shown surprising performance in compressing datasets, there are still several limitations such as distilling high-resolution data or data with complex label spaces. This paper provides a holistic understanding of dataset distillation from multiple aspects, including distillation frameworks and algorithms, factorized dataset distillation, performance comparison, and applications. Finally, we discuss challenges and promising directions to further promote future studies on dataset distillation.
References in corpus (18)
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Condensing Graphs via One-Step Gradient Matching
- Distilled One-Shot Federated Learning
- Soft-Label Anonymous Gastric X-ray Image Distillation
- Compressed Gastric Image Generation Based on Soft-Label Dataset Distillation for Medical Data Sharing
- Federated Learning via Synthetic Data
- FedSynth: Gradient Compression via Synthetic Data in Federated Learning
- Dataset Distillation for Medical Dataset Sharing
- Dataset Condensation with Latent Space Knowledge Factorization and Sharing
- Can we achieve robustness from data alone?
- Data Distillation for Text Classification
- Towards Robust Dataset Learning
- Speeding up NAS with Adaptive Subset Selection
- Meta Knowledge Condensation for Federated Learning
- DiM: Distilling Dataset into Generative Model
- No Free Lunch in "Privacy for Free: How does Dataset Condensation Help Privacy"
- Differentially Private Kernel Inducing Points using features from ScatterNets (DP-KIP-ScatterNet) for Privacy Preserving Data Distillation
- Dataset Condensation via Generative Model