NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision

Abstract

Mobile vision systems such as smartphones, drones, and augmented-reality headsets are revolutionizing our lives. These systems usually run multiple applications concurrently and their available resources at runtime are dynamic due to events such as starting new applications, closing existing applications, and application priority changes. In this paper, we present NestDNN, a framework that takes the dynamics of runtime resources into account to enable resource-aware multi-tenant on-device deep learning for mobile vision systems. NestDNN enables each deep learning model to offer flexible resource-accuracy trade-offs. At runtime, it dynamically selects the optimal resource-accuracy trade-off for each deep learning model to fit the model's resource demand to the system's available runtime resources. In doing so, NestDNN efficiently utilizes the limited resources in mobile vision systems to jointly maximize the performance of all the concurrently running applications. Our experiments show that compared to the resource-agnostic status quo approach, NestDNN achieves as much as 4.2% increase in inference accuracy, 2.0x increase in video frame processing rate and 1.7x reduction on energy consumption.

arXiv comment12 pages

References in corpus (1)

• DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation

Cited by in corpus (18)

• Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
• SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud
• Knowledge Distillation in Deep Learning and its Applications
• Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions
• A Survey on Collaborative DNN Inference for Edge Intelligence
• Artificial Intelligence of Things: A Survey
• Real-time Neural Network Inference on Extremely Weak Devices: Agile Offloading with Explainable AI
• LegoDNN: Block-grained Scaling of Deep Neural Networks for Mobile Vision
• Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques
• On-device Training: A First Overview on Existing Systems
• From Server-Based to Client-Based Machine Learning: A Comprehensive Survey
• Towards Transmission-Friendly and Robust CNN Models over Cloud and Device
• STI: Turbocharge NLP Inference at the Edge via Elastic Pipelining
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Collaborative Inference in DNN-based Satellite Systems with Dynamic Task Streams
• Loki: A System for Serving ML Inference Pipelines with Hardware and Accuracy Scaling
• NestQuant: Post-Training Integer-Nesting Quantization for On-Device DNN
• Elastoformer: Enabling Dynamic Adaptivity via Elastic Model Transformation