Selection via Proxy: Efficient Data Selection for Deep Learning

Abstract

Data selection methods, such as active learning and core-set selection, are useful tools for machine learning on large datasets. However, they can be prohibitively expensive to apply in deep learning because they depend on feature representations that need to be learned. In this work, we show that we can greatly improve the computational efficiency by using a small proxy model to perform data selection (e.g., selecting data points to label for active learning). By removing hidden layers from the target model, using smaller architectures, and training for fewer epochs, we create proxies that are an order of magnitude faster to train. Although these small proxy models have higher error rates, we find that they empirically provide useful signals for data selection. We evaluate this "selection via proxy" (SVP) approach on several data selection tasks across five datasets: CIFAR10, CIFAR100, ImageNet, Amazon Review Polarity, and Amazon Review Full. For active learning, applying SVP can give an order of magnitude improvement in data selection runtime (i.e., the time it takes to repeatedly train and select points) without significantly increasing the final error (often within 0.1%). For core-set selection on CIFAR10, proxies that are over 10x faster to train than their larger, more accurate targets can remove up to 50% of the data without harming the final accuracy of the target, leading to a 1.6x end-to-end training time improvement.

ICLR 2020

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Wide Residual Networks
• Mixed Precision Training
• Exploring the Limits of Language Modeling
• Bayesian Active Learning for Classification and Preference Learning
• Text Understanding from Scratch
• BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning
• Coresets for Scalable Bayesian Logistic Regression
• An Empirical Study of Example Forgetting during Deep Neural Network Learning
• Automated Scalable Bayesian Inference via Hilbert Coresets
• On the Relationship between Data Efficiency and Error for Uncertainty Sampling

Cited by in corpus (12)

• Coresets via Bilevel Optimization for Continual Learning and Streaming
• Self-Supervised Pre-Training for Transformer-Based Person Re-Identification
• On Statistical Bias In Active Learning: How and When To Fix It
• Model Assertions for Monitoring and Improving ML Models
• Minority Class Oriented Active Learning for Imbalanced Datasets
• Improved Synthetic Training for Reading Comprehension
• Data Summarization via Bilevel Optimization
• DIVA: Dataset Derivative of a Learning Task
• Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering
• Accelerating Neural Architecture Search via Proxy Data
• Face-NMS: A Core-set Selection Approach for Efficient Face Recognition
• Accelerating Deep Learning with Dynamic Data Pruning