Auto-Split: A General Framework of Collaborative Edge-Cloud AI

Abstract

In many industry scale applications, large and resource consuming machine learning models reside in powerful cloud servers. At the same time, large amounts of input data are collected at the edge of cloud. The inference results are also communicated to users or passed to downstream tasks at the edge. The edge often consists of a large number of low-power devices. It is a big challenge to design industry products to support sophisticated deep model deployment and conduct model inference in an efficient manner so that the model accuracy remains high and the end-to-end latency is kept low. This paper describes the techniques and engineering practice behind Auto-Split, an edge-cloud collaborative prototype of Huawei Cloud. This patented technology is already validated on selected applications, is on its way for broader systematic edge-cloud application integration, and is being made available for public use as an automated pipeline service for end-to-end cloud-edge collaborative intelligence deployment. To the best of our knowledge, there is no existing industry product that provides the capability of Deep Neural Network (DNN) splitting.

arXiv commentPublished in KDD 2021. Code and demo available at: https://marketplace.huaweicloud.com/markets/aihub/notebook/detail/?id=5fad1eb4-50b2-4ac9-bcb0-a1f744cf85c7

References in corpus (15)

• Distilling the Knowledge in a Neural Network
• Language Models are Few-Shot Learners
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Channel Pruning for Accelerating Very Deep Neural Networks
• SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud
• Model compression via distillation and quantization
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
• Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
• Adaptive Feeding: Achieving Fast and Accurate Detections by Adaptively Combining Object Detectors