Zero Time Waste: Recycling Predictions in Early Exit Neural Networks

Abstract

The problem of reducing processing time of large deep learning models is a fundamental challenge in many real-world applications. Early exit methods strive towards this goal by attaching additional Internal Classifiers (ICs) to intermediate layers of a neural network. ICs can quickly return predictions for easy examples and, as a result, reduce the average inference time of the whole model. However, if a particular IC does not decide to return an answer early, its predictions are discarded, with its computations effectively being wasted. To solve this issue, we introduce Zero Time Waste (ZTW), a novel approach in which each IC reuses predictions returned by its predecessors by (1) adding direct connections between ICs and (2) combining previous outputs in an ensemble-like manner. We conduct extensive experiments across various datasets and architectures to demonstrate that ZTW achieves a significantly better accuracy vs. inference time trade-off than other recently proposed early exit methods.

arXiv commentAccepted at NeurIPS 2021

References in corpus (14)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Wide Residual Networks
• A Survey of Deep Learning Techniques for Autonomous Driving
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Deep Ensembles: A Loss Landscape Perspective
• Challenges of Real-World Reinforcement Learning
• Conditional Computation in Neural Networks for faster models
• Deep Mixture of Experts via Shallow Embedding
• BERT Loses Patience: Fast and Robust Inference with Early Exit
• PoPS: Policy Pruning and Shrinking for Deep Reinforcement Learning
• Dynamically Sacrificing Accuracy for Reduced Computation: Cascaded Inference Based on Softmax Confidence
• GST: Group-Sparse Training for Accelerating Deep Reinforcement Learning

Cited by in corpus (4)

• Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges
• Exploring Dynamic Transformer for Efficient Object Tracking
• Conditional computation in neural networks: principles and research trends
• Dynamic nsNet2: Efficient Deep Noise Suppression with Early Exiting