On-device Training: A First Overview on Existing Systems
arXiv:2212.00824 · doi:10.1145/3696003
Abstract
The recent breakthroughs in machine learning (ML) and deep learning (DL) have catalyzed the design and development of various intelligent systems over wide application domains. While most existing machine learning models require large memory and computing power, efforts have been made to deploy some models on resource-constrained devices as well. A majority of the early application systems focused on exploiting the inference capabilities of ML and DL models, where data captured from different mobile and embedded sensing components are processed through these models for application goals such as classification and segmentation. More recently, the concept of exploiting the mobile and embedded computing resources for ML/DL model training has gained attention, as such capabilities allow (i) the training of models via local data without the need to share data over wireless links, thus enabling privacy-preserving computation by design, (ii) model personalization and environment adaptation, and (ii) deployment of accurate models in remote and hardly accessible locations without stable internet connectivity. This work targets to summarize and analyze state-of-the-art systems research that allows such on-device model training capabilities and provide a survey of on-device training from a systems perspective.
This paper has been accepted to ACM Transactions on Sensor Networks (TOSN), September 2024. DOI: https://doi.org/10.1145/3696003
References in corpus (31)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Neural Architecture Search with Reinforcement Learning
- Knowledge Distillation: A Survey
- BinaryConnect: Training Deep Neural Networks with binary weights during propagations
- Pruning Convolutional Neural Networks for Resource Efficient Inference
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Fast Graph Representation Learning with PyTorch Geometric
- Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
- Towards Accurate Binary Convolutional Neural Network
- SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud
- ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
- MCUNet: Tiny Deep Learning on IoT Devices
- GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
- NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision
- Machine Learning for Microcontroller-Class Hardware: A Review
- DeepCache: Principled Cache for Mobile Deep Vision
- Glow: Graph Lowering Compiler Techniques for Neural Networks
- Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference
- Approximate Query Service on Autonomous IoT Cameras
- Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization
- ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection
- Meta-Learning Representations for Continual Learning
- Accelerating Training and Inference of Graph Neural Networks with Fast Sampling and Pipelining
- Cost-effective On-device Continual Learning over Memory Hierarchy with Miro
- A Graph Theoretic Framework of Recomputation Algorithms for Memory-Efficient Backpropagation
- p-Meta: Towards On-device Deep Model Adaptation
- CGaP: Continuous Growth and Pruning for Efficient Deep Learning
- MNN: A Universal and Efficient Inference Engine