Dynamically Sacrificing Accuracy for Reduced Computation: Cascaded Inference Based on Softmax Confidence

Abstract

We study the tradeoff between computational effort and classification accuracy in a cascade of deep neural networks. During inference, the user sets the acceptable accuracy degradation which then automatically determines confidence thresholds for the intermediate classifiers. As soon as the confidence threshold is met, inference terminates immediately without having to compute the output of the complete network. Confidence levels are derived directly from the softmax outputs of intermediate classifiers, as we do not train special decision functions. We show that using a softmax output as a confidence measure in a cascade of deep neural networks leads to a reduction of 15%-50% in the number of MAC operations while degrading the classification accuracy by roughly 1%. Our method can be easily incorporated into pre-trained non-cascaded architectures, as we exemplify on ResNet. Our main contribution is a method that dynamically adjusts the tradeoff between accuracy and computation without retraining the model.

References in corpus (11)

• On Calibration of Modern Neural Networks
• Adaptive Computation Time for Recurrent Neural Networks
• Training and Inference with Integers in Deep Neural Networks
• Selective Classification for Deep Neural Networks
• EIE: Efficient Inference Engine on Compressed Deep Neural Network
• Adaptive Neural Networks for Efficient Inference
• Spatially Adaptive Computation Time for Residual Networks
• Changing Model Behavior at Test-Time Using Reinforcement Learning
• SGAD: Soft-Guided Adaptively-Dropped Neural Network
• Conditional Deep Learning for Energy-Efficient and Enhanced Pattern Recognition
• URNet : User-Resizable Residual Networks with Conditional Gating Module

Cited by in corpus (5)

• Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions
• Zero Time Waste: Recycling Predictions in Early Exit Neural Networks
• PTEENet: Post-Trained Early-Exit Neural Networks Augmentation for Inference Cost Optimization
• EBJR: Energy-Based Joint Reasoning for Adaptive Inference
• ENAS4D: Efficient Multi-stage CNN Architecture Search for Dynamic Inference