When in Doubt, Summon the Titans: Efficient Inference with Large Models

Abstract

Scaling neural networks to "large" sizes, with billions of parameters, has been shown to yield impressive results on many challenging problems. However, the inference cost incurred by such large models often prevents their application in most real-world settings. In this paper, we propose a two-stage framework based on distillation that realizes the modelling benefits of the large models, while largely preserving the computational benefits of inference with more lightweight models. In a nutshell, we use the large teacher models to guide the lightweight student models to only make correct predictions on a subset of "easy" examples; for the "hard" examples, we fall-back to the teacher. Such an approach allows us to efficiently employ large models in practical scenarios where easy examples are much more frequent than rare hard examples. Our proposed use of distillation to only handle easy instances allows for a more aggressive trade-off in the student size, thereby reducing the amortized cost of inference and achieving better accuracy than standard distillation. Empirically, we demonstrate the benefits of our approach on both image classification and natural language processing benchmarks.

References in corpus (16)

• Distilling the Knowledge in a Neural Network
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• FitNets: Hints for Thin Deep Nets
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Adaptive Computation Time for Recurrent Neural Networks
• Reducing Transformer Depth on Demand with Structured Dropout
• Hierarchical Multiscale Recurrent Neural Networks
• The Devil is in the Tails: Fine-grained Classification in the Wild
• Policy Distillation
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• IDK Cascades: Fast Deep Learning by Learning not to Overthink
• Model compression as constrained optimization, with application to neural nets. Part I: general framework
• Why distillation helps: a statistical perspective
• Controlling Computation versus Quality for Neural Sequence Models
• On the Calibration of Multiclass Classification with Rejection