Shallow-Deep Networks: Understanding and Mitigating Network Overthinking

Abstract

We characterize a prevalent weakness of deep neural networks (DNNs)---overthinking---which occurs when a DNN can reach correct predictions before its final layer. Overthinking is computationally wasteful, and it can also be destructive when, by the final layer, a correct prediction changes into a misclassification. Understanding overthinking requires studying how each prediction evolves during a DNN's forward pass, which conventionally is opaque. For prediction transparency, we propose the Shallow-Deep Network (SDN), a generic modification to off-the-shelf DNNs that introduces internal classifiers. We apply SDN to four modern architectures, trained on three image classification tasks, to characterize the overthinking problem. We show that SDNs can mitigate the wasteful effect of overthinking with confidence-based early exits, which reduce the average inference cost by more than 50% and preserve the accuracy. We also find that the destructive effect occurs for 50% of misclassifications on natural inputs and that it can be induced, adversarially, with a recent backdooring attack. To mitigate this effect, we propose a new confusion metric to quantify the internal disagreements that will likely lead to misclassifications.

arXiv commentAccepted to ICML2019. Source code here: www.shallowdeep.network

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Wide Residual Networks
• On Calibration of Modern Neural Networks
• BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
• Adaptive Neural Networks for Efficient Inference
• IDK Cascades: Fast Deep Learning by Learning not to Overthink
• Greedy Layerwise Learning Can Scale to ImageNet
• One Bit Matters: Understanding Adversarial Examples as the Abuse of Redundancy

Cited by in corpus (12)

• SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud
• Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions
• MobiSR: Efficient On-Device Super-Resolution through Heterogeneous Mobile Processors
• LGViT: Dynamic Early Exiting for Accelerating Vision Transformer
• It's always personal: Using Early Exits for Efficient On-Device CNN Personalisation
• An Embarrassingly Simple Model for Dialogue Relation Extraction
• E${}^2$CM: Early Exit via Class Means for Efficient Supervised and Unsupervised Learning
• One Person, One Model--Learning Compound Router for Sequential Recommendation
• Dynamic nsNet2: Efficient Deep Noise Suppression with Early Exiting
• Latent Iterative Refinement for Modular Source Separation
• Thought Flow Nets: From Single Predictions to Trains of Model Thought
• C2G-Net: Exploiting Morphological Properties for Image Classification