A Survey of Large-Scale Deep Learning Serving System Optimization: Challenges and Opportunities
arXiv:2111.14247
Abstract
Deep Learning (DL) models have achieved superior performance in many application domains, including vision, language, medical, commercial ads, entertainment, etc. With the fast development, both DL applications and the underlying serving hardware have demonstrated strong scaling trends, i.e., Model Scaling and Compute Scaling, for example, the recent pre-trained model with hundreds of billions of parameters with ~TB level memory consumption, as well as the newest GPU accelerators providing hundreds of TFLOPS. With both scaling trends, new problems and challenges emerge in DL inference serving systems, which gradually trends towards Large-scale Deep learning Serving systems (LDS). This survey aims to summarize and categorize the emerging challenges and optimization opportunities for large-scale deep learning serving systems. By providing a novel taxonomy, summarizing the computing paradigms, and elaborating the recent technique advances, we hope that this survey could shed light on new optimization perspectives and motivate novel works in large-scale deep learning system optimization.
10 pages, 7 figures
References in corpus (13)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- Language Models are Few-Shot Learners
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Deep Learning Recommendation Model for Personalization and Recommendation Systems
- Use of Deep Learning in Modern Recommendation System: A Summary of Recent Works
- EmBench: Quantifying Performance Variations of Deep Neural Networks across Modern Commodity Devices
- Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks
- Synergy: A HW/SW Framework for High Throughput CNNs on Embedded Heterogeneous SoC
- Cortex: A Compiler for Recursive Deep Learning Models
- Multi-model Machine Learning Inference Serving with GPU Spatial Partitioning
- IOS: Inter-Operator Scheduler for CNN Acceleration