TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning

Abstract

On-device learning enables edge devices to continually adapt the AI models to new data, which requires a small memory footprint to fit the tight memory constraint of edge devices. Existing work solves this problem by reducing the number of trainable parameters. However, this doesn't directly translate to memory saving since the major bottleneck is the activations, not parameters. In this work, we present Tiny-Transfer-Learning (TinyTL) for memory-efficient on-device learning. TinyTL freezes the weights while only learns the bias modules, thus no need to store the intermediate activations. To maintain the adaptation capacity, we introduce a new memory-efficient bias module, the lite residual module, to refine the feature extractor by learning small residual feature maps adding only 3.8% memory overhead. Extensive experiments show that TinyTL significantly saves the memory (up to 6.5x) with little accuracy loss compared to fine-tuning the full network. Compared to fine-tuning the last layer, TinyTL provides significant accuracy improvements (up to 34.1%) with little memory overhead. Furthermore, combined with feature extractor adaptation, TinyTL provides 7.3-12.9x memory saving without sacrificing accuracy compared to fine-tuning the full Inception-V3.

NeurIPS 2020

References in corpus (8)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Compressing Deep Convolutional Networks using Vector Quantization
• Efficient Architecture Search by Network Transformation
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• Highway and Residual Networks learn Unrolled Iterative Estimation
• Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
• BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
• RNNPool: Efficient Non-linear Pooling for RAM Constrained Inference

Cited by in corpus (11)

• Sustainable AI: Environmental Implications, Challenges and Opportunities
• Tiny Machine Learning: Progress and Futures
• Bringing AI To Edge: From Deep Learning's Perspective
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection
• Federated Few-Shot Learning for Mobile NLP
• Intelligent Model Update Strategy for Sequential Recommendation
• Rock Hunting With Martian Machine Vision
• TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search
• Advancing On-Device Neural Network Training with TinyPropv2: Dynamic, Sparse, and Efficient Backpropagation
• MixMix: All You Need for Data-Free Compression Are Feature and Data Mixing