Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Model

Abstract

We study how to train a student deep neural network for visual recognition by distilling knowledge from a blackbox teacher model in a data-efficient manner. Progress on this problem can significantly reduce the dependence on large-scale datasets for learning high-performing visual recognition models. There are two major challenges. One is that the number of queries into the teacher model should be minimized to save computational and/or financial costs. The other is that the number of images used for the knowledge distillation should be small; otherwise, it violates our expectation of reducing the dependence on large-scale datasets. To tackle these challenges, we propose an approach that blends mixup and active learning. The former effectively augments the few unlabeled images by a big pool of synthetic images sampled from the convex hull of the original images, and the latter actively chooses from the pool hard examples for the student neural network and query their labels from the teacher model. We validate our approach with extensive experiments.

arXiv commentAccepted to CVPR 2020

References in corpus (19)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• mixup: Beyond Empirical Risk Minimization
• ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• AutoAugment: Learning Augmentation Policies from Data
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• Data Augmentation by Pairing Samples for Images Classification
• Black-box Adversarial Attacks with Limited Queries and Information
• Data-Free Knowledge Distillation for Deep Neural Networks
• Learning from Between-class Examples for Deep Sound Recognition
• Adversarial Active Learning for Deep Networks: a Margin Based Approach
• A Benchmark and Comparison of Active Learning for Logistic Regression
• Discriminative Active Learning
• Zero-Shot Knowledge Distillation in Deep Networks
• Few-shot learning of neural networks from scratch by pseudo example optimization
• Knowledge Distillation via Route Constrained Optimization
• A Comprehensive Overhaul of Feature Distillation