ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection
arXiv:2312.14227 · doi:10.1145/3581791.3596852
Abstract
On-device training is essential for neural networks (NNs) to continuously adapt to new online data, but can be time-consuming due to the device's limited computing power. To speed up on-device training, existing schemes select trainable NN portion offline or conduct unrecoverable selection at runtime, but the evolution of trainable NN portion is constrained and cannot adapt to the current need for training. Instead, runtime adaptation of on-device training should be fully elastic, i.e., every NN substructure can be freely removed from or added to the trainable NN portion at any time in training. In this paper, we present ElasticTrainer, a new technique that enforces such elasticity to achieve the required training speedup with the minimum NN accuracy loss. Experiment results show that ElasticTrainer achieves up to 3.5x more training speedup in wall-clock time and reduces energy consumption by 2x-3x more compared to the existing schemes, without noticeable accuracy loss.
arXiv commentpublished at ACM MobiSys 2023. 14 pages
References in corpus (22)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Neural Machine Translation by Jointly Learning to Align and Translate
- Decoupled Weight Decay Regularization
- Overcoming catastrophic forgetting in neural networks
- A Brief Survey of Deep Reinforcement Learning
- DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
- Axiomatic Attribution for Deep Networks
- LoRA: Low-Rank Adaptation of Large Language Models
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- Towards Federated Learning at Scale: System Design
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- Evaluation of Output Embeddings for Fine-Grained Image Classification
- meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
- Split Learning for collaborative deep learning in healthcare
- On-Device Training Under 256KB Memory
- PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfiguration
- K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning
- TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning
- Mandheling: Mixed-Precision On-Device DNN Training with DSP Offloading