Real-time Neural Network Inference on Extremely Weak Devices: Agile Offloading with Explainable AI

Abstract

With the wide adoption of AI applications, there is a pressing need of enabling real-time neural network (NN) inference on small embedded devices, but deploying NNs and achieving high performance of NN inference on these small devices is challenging due to their extremely weak capabilities. Although NN partitioning and offloading can contribute to such deployment, they are incapable of minimizing the local costs at embedded devices. Instead, we suggest to address this challenge via agile NN offloading, which migrates the required computations in NN offloading from online inference to offline learning. In this paper, we present AgileNN, a new NN offloading technique that achieves real-time NN inference on weak embedded devices by leveraging eXplainable AI techniques, so as to explicitly enforce feature sparsity during the training phase and minimize the online computation and communication costs. Experiment results show that AgileNN's inference latency is >6x lower than the existing schemes, ensuring that sensory data on embedded devices can be timely consumed. It also reduces the local device's resource consumption by >8x, without impairing the inference accuracy.

arXiv commentpublished at ACM MobiCom 2022. 14 pages

References in corpus (21)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Machine Translation by Jointly Learning to Align and Translate
• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• EfficientNetV2: Smaller Models and Faster Training
• Compressing Deep Convolutional Networks using Vector Quantization
• Model-Agnostic Interpretability of Machine Learning
• Deep Learning in Video Multi-Object Tracking: A Survey
• GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
• Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations
• SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud
• MCUNet: Tiny Deep Learning on IoT Devices
• NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision
• PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruning
• JALAD: Joint Accuracy- and Latency-Aware Deep Structure Decoupling for Edge-Cloud Execution
• MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers
• Intention-Net: Integrating Planning and Deep Learning for Goal-Directed Autonomous Navigation
• Pruning Neural Networks at Initialization: Why are We Missing the Mark?
• Intelligence Beyond the Edge: Inference on Intermittent Embedded Systems
• DeepN-JPEG: A Deep Neural Network Favorable JPEG-based Image Compression Framework

Cited by in corpus (7)

• Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models
• Edge Deep Learning in Computer Vision and Medical Diagnostics: A Comprehensive Survey
• Artificial Intelligence of Things: A Survey
• SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget
• LimitNet: Progressive, Content-Aware Image Offloading for Extremely Weak Devices & Networks
• Semantic Edge Computing and Semantic Communications in 6G Networks: A Unifying Survey and Research Challenges
• da4ml: Distributed Arithmetic for Real-time Neural Networks on FPGAs