MCUNet: Tiny Deep Learning on IoT Devices

Abstract

Machine learning on tiny IoT devices based on microcontroller units (MCU) is appealing but challenging: the memory of microcontrollers is 2-3 orders of magnitude smaller even than mobile phones. We propose MCUNet, a framework that jointly designs the efficient neural architecture (TinyNAS) and the lightweight inference engine (TinyEngine), enabling ImageNet-scale inference on microcontrollers. TinyNAS adopts a two-stage neural architecture search approach that first optimizes the search space to fit the resource constraints, then specializes the network architecture in the optimized search space. TinyNAS can automatically handle diverse constraints (i.e.device, latency, energy, memory) under low search costs.TinyNAS is co-designed with TinyEngine, a memory-efficient inference library to expand the search space and fit a larger model. TinyEngine adapts the memory scheduling according to the overall network topology rather than layer-wise optimization, reducing the memory usage by 4.8x, and accelerating the inference by 1.7-3.3x compared to TF-Lite Micro and CMSIS-NN. MCUNet is the first to achieves >70% ImageNet top1 accuracy on an off-the-shelf commercial microcontroller, using 3.5x less SRAM and 5.7x less Flash compared to quantized MobileNetV2 and ResNet-18. On visual&audio wake words tasks, MCUNet achieves state-of-the-art accuracy and runs 2.4-3.4x faster than MobileNetV2 and ProxylessNAS-based solutions with 3.7-4.1x smaller peak SRAM. Our study suggests that the era of always-on tiny machine learning on IoT devices has arrived. Code and models can be found here: https://tinyml.mit.edu.

arXiv commentNeurIPS 2020 (spotlight)

References in corpus (22)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• TensorFlow: A system for large-scale machine learning
• Neural Architecture Search with Reinforcement Learning
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• SGDR: Stochastic Gradient Descent with Warm Restarts
• DARTS: Differentiable Architecture Search
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Compressing Deep Convolutional Networks using Vector Quantization
• An Analysis of Deep Neural Network Models for Practical Applications
• Trained Ternary Quantization
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Learning Efficient Convolutional Networks through Network Slimming
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Learning to Optimize Tensor Programs
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
• MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
• Visual Wake Words Dataset
• SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers
• Neural networks on microcontrollers: saving memory at inference via operator reordering

Cited by in corpus (34)

• A Review on Deep Learning in UAV Remote Sensing
• Machine Learning for Microcontroller-Class Hardware: A Review
• Quantization and Deployment of Deep Neural Networks on Microcontrollers
• Tiny Machine Learning: Progress and Futures
• Bringing AI To Edge: From Deep Learning's Perspective
• TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems
• MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• Environmental Sound Classification on the Edge: A Pipeline for Deep Acoustic Networks on Extremely Resource-Constrained Devices
• Real-time Neural Network Inference on Extremely Weak Devices: Agile Offloading with Explainable AI
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
• An Ultra-low Power TinyML System for Real-time Visual Processing at Edge
• TinyissimoYOLO: A Quantized, Low-Memory Footprint, TinyML Object Detection Network for Low Power Microcontrollers
• On-device Training: A First Overview on Existing Systems
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• CODEBench: A Neural Architecture and Hardware Accelerator Co-Design Framework
• Mixed-precision Neural Networks on RISC-V Cores: ISA extensions for Multi-Pumped Soft SIMD Operations
• DASS: Differentiable Architecture Search for Sparse neural networks
• LimitNet: Progressive, Content-Aware Image Offloading for Extremely Weak Devices & Networks
• Pruning In Time (PIT): A Lightweight Network Architecture Optimizer for Temporal Convolutional Networks
• Tiny Transformers for Environmental Sound Classification at the Edge
• Enabling Large Neural Networks on Tiny Microcontrollers with Swapping
• Secure Watermark for Deep Neural Networks with Multi-task Learning
• AnalogNets: ML-HW Co-Design of Noise-robust TinyML Models and Always-On Analog Compute-in-Memory Accelerator
• Running hardware-aware neural architecture search on embedded devices under 512MB of RAM
• EdgeMark: An Automation and Benchmarking System for Embedded Artificial Intelligence Tools
• TinyOL: TinyML with Online-Learning on Microcontrollers
• Dynamic-OFA: Runtime DNN Architecture Switching for Performance Scaling on Heterogeneous Embedded Platforms
• Generic Neural Architecture Search via Regression
• StreamTinyNet: video streaming analysis with spatial-temporal TinyML
• MCUBERT: Memory-Efficient BERT Inference on Commodity Microcontrollers
• Reducing Neural Architecture Search Spaces with Training-Free Statistics and Computational Graph Clustering
• DietCNN: Multiplication-free Inference for Quantized CNNs
• MUDAS: Mote-scale Unsupervised Domain Adaptation in Multi-label Sound Classification