Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions

Abstract

DNNs are becoming less and less over-parametrised due to recent advances in efficient model design, through careful hand-crafted or NAS-based methods. Relying on the fact that not all inputs require the same amount of computation to yield a confident prediction, adaptive inference is gaining attention as a prominent approach for pushing the limits of efficient deployment. Particularly, early-exit networks comprise an emerging direction for tailoring the computation depth of each input sample at runtime, offering complementary performance gains to other efficiency optimisations. In this paper, we decompose the design methodology of early-exit networks to its key components and survey the recent advances in each one of them. We also position early-exiting against other efficient inference solutions and provide our insights on the current challenges and most promising future directions for research in the field.

arXiv commentAccepted at the 5th Annual International Workshop on Embedded and Mobile Deep Learning (EMDL), 2021

References in corpus (38)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Deep Residual Learning for Image Recognition
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Deep Learning for Sensor-based Activity Recognition: A Survey
• On Calibration of Modern Neural Networks
• Ternary Weight Networks
• Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges
• SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud
• To Trust Or Not To Trust A Classifier
• NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• Slimmable Neural Networks
• Do Convolutional Neural Networks Learn Class Hierarchy?
• Dynamic Channel Pruning: Feature Boosting and Suppression
• Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
• Dual Dynamic Inference: Enabling More Efficient, Adaptive and Controllable Deep Inference
• MobiSR: Efficient On-Device Super-Resolution through Heterogeneous Mobile Processors
• Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
• EmBench: Quantifying Performance Variations of Deep Neural Networks across Modern Commodity Devices
• FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
• Depth-Adaptive Transformer
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• Early Exit or Not: Resource-Efficient Blind Quality Enhancement for Compressed Images
• Pruning Algorithms to Accelerate Convolutional Neural Networks for Edge Applications: A Survey
• Dynamic Neural Networks: A Survey
• Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference
• HAPI: Hardware-Aware Progressive Inference
• IDK Cascades: Fast Deep Learning by Learning not to Overthink
• Dynamically Sacrificing Accuracy for Reduced Computation: Cascaded Inference Based on Softmax Confidence
• Not All Pixels Are Equal: Difficulty-aware Semantic Segmentation via Deep Layer Cascade
• It's always personal: Using Early Exits for Efficient On-Device CNN Personalisation
• Learning to Stop While Learning to Predict
• A Panda? No, It's a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inference
• Temporally Distributed Networks for Fast Video Semantic Segmentation
• Conditional Deep Learning for Energy-Efficient and Enhanced Pattern Recognition
• Fractional Skipping: Towards Finer-Grained Dynamic CNN Inference

Cited by in corpus (8)

• Mobile Foundation Model as Firmware
• DynO: Dynamic Onloading of Deep Neural Networks from Cloud to Device
• Dynamic nsNet2: Efficient Deep Noise Suppression with Early Exiting
• Tiny Models are the Computational Saver for Large Models
• Early Exiting Predictive Coding Neural Networks for Edge AI
• Hetero-SplitEE: Split Learning of Neural Networks with Early Exits for Heterogeneous IoT Devices
• Spike-inspired Rank Coding for Fast and Accurate Recurrent Neural Networks
• Elastoformer: Enabling Dynamic Adaptivity via Elastic Model Transformation