Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications

Abstract

Deep neural networks (DNNs) have achieved unprecedented success in the field of artificial intelligence (AI), including computer vision, natural language processing and speech recognition. However, their superior performance comes at the considerable cost of computational complexity, which greatly hinders their applications in many resource-constrained devices, such as mobile phones and Internet of Things (IoT) devices. Therefore, methods and techniques that are able to lift the efficiency bottleneck while preserving the high accuracy of DNNs are in great demand in order to enable numerous edge AI applications. This paper provides an overview of efficient deep learning methods, systems and applications. We start from introducing popular model compression methods, including pruning, factorization, quantization as well as compact model design. To reduce the large design cost of these manual solutions, we discuss the AutoML framework for each of them, such as neural architecture search (NAS) and automated pruning and quantization. We then cover efficient on-device training to enable user customization based on the local data on mobile devices. Apart from general acceleration techniques, we also showcase several task-specific accelerations for point cloud, video and natural language processing by exploiting their spatial sparsity and temporal/token redundancy. Finally, to support all these algorithmic advancements, we introduce the efficient deep learning system design from both software and hardware perspectives.

arXiv commentJournal preprint (ACM TODAES, 2021). The first seven authors contributed equally to this work and are listed in the alphabetical order

References in corpus (92)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Neural Machine Translation by Jointly Learning to Align and Translate
• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• TensorFlow: A system for large-scale machine learning
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Neural Architecture Search with Reinforcement Learning
• Character-level Convolutional Networks for Text Classification
• End to End Learning for Self-Driving Cars
• Federated Learning: Strategies for Improving Communication Efficiency
• ShapeNet: An Information-Rich 3D Model Repository
• Longformer: The Long-Document Transformer
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• FitNets: Hints for Thin Deep Nets
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Neural Architecture Search: A Survey
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• AutoML: A Survey of the State-of-the-Art
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Fine-Grained Visual Classification of Aircraft
• Deep Learning with Limited Numerical Precision
• FINN: A Framework for Fast, Scalable Binarized Neural Network Inference
• Compressing Deep Convolutional Networks using Vector Quantization
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Linformer: Self-Attention with Linear Complexity
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• O-CNN: Octree-based Convolutional Neural Networks for 3D Shape Analysis
• Ternary Weight Networks
• Trained Ternary Quantization
• Joint 2D-3D-Semantic Data for Indoor Scene Understanding
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Training Deep Nets with Sublinear Memory Cost
• Non-Autoregressive Neural Machine Translation
• Spatiotemporal Residual Networks for Video Action Recognition
• The Microsoft 2017 Conversational Speech Recognition System
• SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• Point-Voxel CNN for Efficient 3D Deep Learning
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• GCN-RL Circuit Designer: Transferable Transistor Sizing with Graph Neural Networks and Reinforcement Learning
• MCUNet: Tiny Deep Learning on IoT Devices
• Slimmable Neural Networks
• Efficient Transformers: A Survey
• Device Placement Optimization with Reinforcement Learning
• HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
• DeepGCNs: Making GCNs Go as Deep as CNNs
• PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruning
• A Survey on Neural Architecture Search
• Gradient Sparsification for Communication-Efficient Distributed Optimization
• PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection
• Levenshtein Transformer
• DetNAS: Backbone Search for Object Detection
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• Path-Level Network Transformation for Efficient Architecture Search
• MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• EIE: Efficient Inference Engine on Compressed Deep Neural Network
• Faster CNNs with Direct Sparse Convolutions and Guided Pruning
• SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks
• Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
• Adaptive O-CNN: A Patch-based Deep Representation of 3D Shapes
• Linear Mode Connectivity and the Lottery Ticket Hypothesis
• More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation
• SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers
• Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
• Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research
• Syntactically Supervised Transformers for Faster Neural Machine Translation
• PVNAS: 3D Neural Architecture Search with Point-Voxel Convolution
• Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes
• BinaryBERT: Pushing the Limit of BERT Quantization
• K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning
• TorchSparse: Efficient Point Cloud Inference Engine
• AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations
• Hardware-Centric AutoML for Mixed-Precision Quantization
• Blockwise Self-Attention for Long Document Understanding
• A3D: Adaptive 3D Networks for Video Action Recognition
• Training Kinetics in 15 Minutes: Large-scale Distributed Training on Videos
• Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
• PCNN: Pattern-based Fine-Grained Regular Pruning towards Optimizing CNN Accelerators
• PV-NAS: Practical Neural Architecture Search for Video Recognition
• SparseDNN: Fast Sparse Deep Learning Inference on CPUs
• Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning
• AutoADR: Automatic Model Design for Ad Relevance

Cited by in corpus (7)

• Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models
• Artificial Intelligence of Things: A Survey
• Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey
• Enhancing Neural Architecture Search with Multiple Hardware Constraints for Deep Learning Model Deployment on Tiny IoT Devices
• Real-time Short Video Recommendation on Mobile Devices
• Peer-to-Peer Learning Dynamics of Wide Neural Networks
• Neural network scoring for efficient computing