On-Device Machine Learning: An Algorithms and Learning Theory Perspective

Abstract

The predominant paradigm for using machine learning models on a device is to train a model in the cloud and perform inference using the trained model on the device. However, with increasing number of smart devices and improved hardware, there is interest in performing model training on the device. Given this surge in interest, a comprehensive survey of the field from a device-agnostic perspective sets the stage for both understanding the state-of-the-art and for identifying open challenges and future avenues of research. However, on-device learning is an expansive field with connections to a large number of related topics in AI and machine learning (including online learning, model adaptation, one/few-shot learning, etc.). Hence, covering such a large number of topics in a single survey is impractical. This survey finds a middle ground by reformulating the problem of on-device learning as resource constrained learning where the resources are compute and memory. This reformulation allows tools, techniques, and algorithms from a wide variety of research areas to be compared equitably. In addition to summarizing the state-of-the-art, the survey also identifies a number of challenges and next steps for both the algorithmic and theoretical aspects of on-device learning.

Edge Learning, TinyML, Resource Constrained Machine Learning, Deep learning on device, Statistical Learning Theory, 45 pages survey

References in corpus (61)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Improving neural networks by preventing co-adaptation of feature detectors
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• A Survey of Deep Learning-based Object Detection
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Recent Advances in Open Set Recognition: A Survey
• Compressing Deep Convolutional Networks using Vector Quantization
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Mixed Precision Training
• An Analysis of Deep Neural Network Models for Practical Applications
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Benchmark Analysis of Representative Deep Neural Network Architectures
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Compressing Neural Networks with the Hashing Trick
• Training Deep Nets with Sublinear Memory Cost
• MixConv: Mixed Depthwise Convolutional Kernels
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• Sparsified SGD with Memory
• A Survey on Neural Architecture Search
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• A Survey on Methods and Theories of Quantized Neural Networks
• Fathom: Reference Workloads for Modern Deep Learning Methods
• Communication Compression for Decentralized Training
• Bitwise Neural Networks
• Neural Networks with Few Multiplications
• Adaptive Neural Networks for Efficient Inference
• Training with Quantization Noise for Extreme Model Compression
• Highway and Residual Networks learn Unrolled Iterative Estimation
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• NeuralPower: Predict and Deploy Energy-Efficient Convolutional Neural Networks
• Fundamental Limits of Online and Distributed Algorithms for Statistical Learning and Estimation
• Modeling the Resource Requirements of Convolutional Neural Networks on Mobile Devices
• High-Accuracy Low-Precision Training
• Memory Limited, Streaming PCA
• Efficient Learning with Partially Observed Attributes
• TBD: Benchmarking and Analyzing Deep Neural Network Training
• Memory-Efficient Backpropagation Through Time
• SkipNet: Learning Dynamic Routing in Convolutional Networks
• ProjectionNet: Learning Efficient On-Device Deep Networks Using Neural Projections
• LightRNN: Memory and Computation-Efficient Recurrent Neural Networks
• Pruning Algorithms to Accelerate Convolutional Neural Networks for Edge Applications: A Survey
• Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization
• More data speeds up training time in learning halfspaces over sparse vectors
• FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
• BigDataBench: A Scalable and Unified Big Data and AI Benchmark Suite
• The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning
• Large-Scale Learning with Less RAM via Randomization
• Deploying Deep Neural Networks in the Embedded Space
• Time-Space Tradeoffs for Learning from Small Test Spaces: Learning Low Degree Polynomial Functions
• From average case complexity to improper learning complexity
• Sample Efficient Stochastic Gradient Iterative Hard Thresholding Method for Stochastic Sparse Linear Regression with Limited Attribute Observation
• A General Memory-Bounded Learning Algorithm
• On sample complexity for computational pattern recognition
• Fine-Grained Energy and Performance Profiling framework for Deep Convolutional Neural Networks

Cited by in corpus (7)

• Bringing AI To Edge: From Deep Learning's Perspective
• TinyML for Ubiquitous Edge AI
• COVID-MobileXpert: On-Device COVID-19 Patient Triage and Follow-up using Chest X-rays
• SEFR: A Fast Linear-Time Classifier for Ultra-Low Power Devices
• Channel-Driven Monte Carlo Sampling for Bayesian Distributed Learning in Wireless Data Centers
• Asynchronous Federated Learning for Sensor Data with Concept Drift
• LCS: Learning Compressible Subspaces for Adaptive Network Compression at Inference Time