Model Slicing for Supporting Complex Analytics with Elastic Inference Cost and Resource Constraints

Abstract

Deep learning models have been used to support analytics beyond simple aggregation, where deeper and wider models have been shown to yield great results. These models consume a huge amount of memory and computational operations. However, most of the large-scale industrial applications are often computational budget constrained. In practice, the peak workload of inference service could be 10x higher than the average cases, with the presence of unpredictable extreme cases. Lots of computational resources could be wasted during off-peak hours and the system may crash when the workload exceeds system capacity. How to support deep learning services with a dynamic workload cost-efficiently remains a challenging problem. In this paper, we address the challenge with a general and novel training scheme called model slicing, which enables deep learning models to provide predictions within the prescribed computational resource budget dynamically. Model slicing could be viewed as an elastic computation solution without requiring more computational resources. Succinctly, each layer in the model is divided into groups of a contiguous block of basic components (i.e. neurons in dense layers and channels in convolutional layers), and then partially ordered relation is introduced to these groups by enforcing that groups participated in each forward pass always starts from the first group to the dynamically-determined rightmost group. Trained by dynamically indexing the rightmost group with a single parameter slice rate, the network is engendered to build up group-wise and residual representation. Then during inference, a sub-model with fewer groups can be readily deployed for efficiency whose computation is roughly quadratic to the width controlled by the slice rate. Extensive experiments show that models trained with model slicing can effectively support on-demand workload with elastic inference cost.

arXiv comment14 pages, 8 figures

References in corpus (23)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• XGBoost: A Scalable Tree Boosting System
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Recurrent Neural Network Regularization
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Wide Residual Networks
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Pruning Filters for Efficient ConvNets
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• Compressing Neural Networks with the Hashing Trick
• Learning Structured Sparsity in Deep Neural Networks
• Deep Networks with Stochastic Depth
• Slimmable Neural Networks
• Cascade Ranking for Operational E-commerce Search
• The Power of Sparsity in Convolutional Neural Networks
• NoScope: Optimizing Neural Network Queries over Video at Scale
• IDK Cascades: Fast Deep Learning by Learning not to Overthink
• Dynamic Routing Networks

Cited by in corpus (6)

• Fast Parallel Hypertree Decompositions in Logarithmic Recursion Depth
• Database Meets Deep Learning: Challenges and Opportunities
• Effective and Efficient Dropout for Deep Convolutional Neural Networks
• NeurDB: An AI-powered Autonomous Data System
• MLCask: Efficient Management of Component Evolution in Collaborative Data Analytics Pipelines
• LoCEC: Local Community-based Edge Classification in Large Online Social Networks