Dual Dynamic Inference: Enabling More Efficient, Adaptive and Controllable Deep Inference

Abstract

State-of-the-art convolutional neural networks (CNNs) yield record-breaking predictive performance, yet at the cost of high-energy-consumption inference, that prohibits their widely deployments in resource-constrained Internet of Things (IoT) applications. We propose a dual dynamic inference (DDI) framework that highlights the following aspects: 1) we integrate both input-dependent and resource-dependent dynamic inference mechanisms under a unified framework in order to fit the varying IoT resource requirements in practice. DDI is able to both constantly suppress unnecessary costs for easy samples, and to halt inference for all samples to meet hard resource constraints enforced; 2) we propose a flexible multi-grained learning to skip (MGL2S) approach for input-dependent inference which allows simultaneous layer-wise and channel-wise skipping; 3) we extend DDI to complex CNN backbones such as DenseNet and show that DDI can be applied towards optimizing any specific resource goals including inference latency or energy cost. Extensive experiments demonstrate the superior inference accuracy-resource trade-off achieved by DDI, as well as the flexibility to control such trade-offs compared to existing peer methods. Specifically, DDI can achieve up to 4 times computational savings with the same or even higher accuracy as compared to existing competitive baselines.

References in corpus (10)

• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• One Model To Learn Them All
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• Slimmable Neural Networks
• Deep $k$-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions
• Hybrid Pruning: Thinner Sparse Networks for Fast Inference on Edge Devices

Cited by in corpus (13)

• Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges
• Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions
• AutoDNNchip: An Automated DNN Chip Predictor and Builder for Both FPGAs and ASICs
• IA-RED${}^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
• Dynamic Neural Networks: A Survey
• 2-in-1 Accelerator: Enabling Random Precision Switch for Winning Both Adversarial Robustness and Efficiency
• SmartExchange: Trading Higher-cost Memory Storage/Access for Lower-cost Computation
• TIMELY: Pushing Data Movements and Interfaces in PIM Accelerators Towards Local and in Time Domain
• AppealNet: An Efficient and Highly-Accurate Edge/Cloud Collaborative Architecture for DNN Inference
• Graceful Degradation and Related Fields
• CoDiNet: Path Distribution Modeling with Consistency and Diversity for Dynamic Routing
• DNN-Chip Predictor: An Analytical Performance Predictor for DNN Accelerators with Various Dataflows and Hardware Architectures
• SmartDeal: Re-Modeling Deep Network Weights for Efficient Inference and Training