Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions

Abstract

DNNs are becoming less and less over-parametrised due to recent advances in efficient model design, through careful hand-crafted or NAS-based methods. Relying on the fact that not all inputs require the same amount of computation to yield a confident prediction, adaptive inference is gaining attention as a prominent approach for pushing the limits of efficient deployment. Particularly, early-exit networks comprise an emerging direction for tailoring the computation depth of each input sample at runtime, offering complementary performance gains to other efficiency optimisations. In this paper, we decompose the design methodology of early-exit networks to its key components and survey the recent advances in each one of them. We also position early-exiting against other efficient inference solutions and provide our insights on the current challenges and most promising future directions for research in the field.

Accepted at the 5th Annual International Workshop on Embedded and Mobile Deep Learning (EMDL), 2021

References in corpus (16)

• Distilling the Knowledge in a Neural Network
• On Calibration of Modern Neural Networks
• SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud
• Slimmable Neural Networks
• Do Convolutional Neural Networks Learn Class Hierarchy?
• Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
• EmBench: Quantifying Performance Variations of Deep Neural Networks across Modern Commodity Devices
• Depth-Adaptive Transformer
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• Pruning Algorithms to Accelerate Convolutional Neural Networks for Edge Applications: A Survey
• Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference
• HAPI: Hardware-Aware Progressive Inference
• It's always personal: Using Early Exits for Efficient On-Device CNN Personalisation
• Learning to Stop While Learning to Predict
• Temporally Distributed Networks for Fast Video Semantic Segmentation
• Fractional Skipping: Towards Finer-Grained Dynamic CNN Inference