Pipeline Parallelism for Inference on Heterogeneous Edge Computing

Abstract

Deep neural networks with large model sizes achieve state-of-the-art results for tasks in computer vision (CV) and natural language processing (NLP). However, these large-scale models are too compute- or memory-intensive for resource-constrained edge devices. Prior works on parallel and distributed execution primarily focus on training -- rather than inference -- using homogeneous accelerators in data centers. We propose EdgePipe, a distributed framework for edge systems that uses pipeline parallelism to both speed up inference and enable running larger (and more accurate) models that otherwise cannot fit on single edge devices. EdgePipe achieves these results by using an optimal partition strategy that considers heterogeneity in compute, memory, and network bandwidth. Our empirical evaluation demonstrates that EdgePipe achieves $10.59\times$ and $11.88\times$ speedup using 16 edge devices for the ViT-Large and ViT-Huge models, respectively, with no accuracy loss. Similarly, EdgePipe improves ViT-Huge throughput by $3.93\times$ over a 4-node baseline using 16 edge devices, which independently cannot fit the model in memory. Finally, we show up to $4.16\times$ throughput improvement over the state-of-the-art PipeDream when using a heterogeneous set of devices.

References in corpus (12)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
• HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
• Lite Transformer with Long-Short Range Attention
• HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism
• TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models
• torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models
• PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers
• EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inference