TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning

Abstract

On-device learning enables edge devices to continually adapt the AI models to new data, which requires a small memory footprint to fit the tight memory constraint of edge devices. Existing work solves this problem by reducing the number of trainable parameters. However, this doesn't directly translate to memory saving since the major bottleneck is the activations, not parameters. In this work, we present Tiny-Transfer-Learning (TinyTL) for memory-efficient on-device learning. TinyTL freezes the weights while only learns the bias modules, thus no need to store the intermediate activations. To maintain the adaptation capacity, we introduce a new memory-efficient bias module, the lite residual module, to refine the feature extractor by learning small residual feature maps adding only 3.8% memory overhead. Extensive experiments show that TinyTL significantly saves the memory (up to 6.5x) with little accuracy loss compared to fine-tuning the full network. Compared to fine-tuning the last layer, TinyTL provides significant accuracy improvements (up to 34.1%) with little memory overhead. Furthermore, combined with feature extractor adaptation, TinyTL provides 7.3-12.9x memory saving without sacrificing accuracy compared to fine-tuning the full Inception-V3.

arXiv commentNeurIPS 2020

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Empirical Evaluation of Rectified Activations in Convolutional Network
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Fine-Grained Visual Classification of Aircraft
• Compressing Deep Convolutional Networks using Vector Quantization
• Training Deep Nets with Sublinear Memory Cost
• Efficient Architecture Search by Network Transformation
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
• Highway and Residual Networks learn Unrolled Iterative Estimation
• Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
• K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning
• BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
• RNNPool: Efficient Non-linear Pooling for RAM Constrained Inference
• MemNet: Memory-Efficiency Guided Neural Architecture Search with Augment-Trim learning

Cited by in corpus (11)

• Sustainable AI: Environmental Implications, Challenges and Opportunities
• Tiny Machine Learning: Progress and Futures
• Bringing AI To Edge: From Deep Learning's Perspective
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection
• Federated Few-Shot Learning for Mobile NLP
• Intelligent Model Update Strategy for Sequential Recommendation
• Rock Hunting With Martian Machine Vision
• TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search
• Advancing On-Device Neural Network Training with TinyPropv2: Dynamic, Sparse, and Efficient Backpropagation
• MixMix: All You Need for Data-Free Compression Are Feature and Data Mixing