SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud

Abstract

Despite the soaring use of convolutional neural networks (CNNs) in mobile applications, uniformly sustaining high-performance inference on mobile has been elusive due to the excessive computational demands of modern CNNs and the increasing diversity of deployed devices. A popular alternative comprises offloading CNN processing to powerful cloud-based servers. Nevertheless, by relying on the cloud to produce outputs, emerging mission-critical and high-mobility applications, such as drone obstacle avoidance or interactive applications, can suffer from the dynamic connectivity conditions and the uncertain availability of the cloud. In this paper, we propose SPINN, a distributed inference system that employs synergistic device-cloud computation together with a progressive inference method to deliver fast and robust CNN inference across diverse settings. The proposed system introduces a novel scheduler that co-optimises the early-exit policy and the CNN splitting at run time, in order to adapt to dynamic conditions and meet user-defined service-level requirements. Quantitative evaluation illustrates that SPINN outperforms its state-of-the-art collaborative inference counterparts by up to 2x in achieved throughput under varying network conditions, reduces the server cost by up to 6.8x and improves accuracy by 20.7% under latency constraints, while providing robust operation under uncertain connectivity conditions and significant energy savings compared to cloud-centric execution.

Accepted at the 26th Annual International Conference on Mobile Computing and Networking (MobiCom), 2020

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• On Calibration of Modern Neural Networks
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• JALAD: Joint Accuracy- and Latency-Aware Deep Structure Decoupling for Edge-Cloud Execution
• Focus: Querying Large Video Datasets with Low Latency and Low Cost
• EmBench: Quantifying Performance Variations of Deep Neural Networks across Modern Commodity Devices
• HAPI: Hardware-Aware Progressive Inference

Cited by in corpus (10)

• Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions
• It's always personal: Using Early Exits for Efficient On-Device CNN Personalisation
• Smart at what cost? Characterising Mobile Deep Neural Networks in the wild
• ACE: Towards Application-Centric Edge-Cloud Collaborative Intelligence
• FTPipeHD: A Fault-Tolerant Pipeline-Parallel Distributed Training Framework for Heterogeneous Edge Devices
• Edge-Cloud Collaborated Object Detection via Difficult-Case Discriminator
• Deep Learning Techniques for Compressive Sensing-Based Reconstruction and Inference -- A Ubiquitous Systems Perspective
• Complexity-aware Adaptive Training and Inference for Edge-Cloud Distributed AI Systems
• Auto-Split: A General Framework of Collaborative Edge-Cloud AI
• Dynamic DNN Decomposition for Lossless Synergistic Inference