MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning

Abstract

Tiny deep learning on microcontroller units (MCUs) is challenging due to the limited memory size. We find that the memory bottleneck is due to the imbalanced memory distribution in convolutional neural network (CNN) designs: the first several blocks have an order of magnitude larger memory usage than the rest of the network. To alleviate this issue, we propose a generic patch-by-patch inference scheduling, which operates only on a small spatial region of the feature map and significantly cuts down the peak memory. However, naive implementation brings overlapping patches and computation overhead. We further propose network redistribution to shift the receptive field and FLOPs to the later stage and reduce the computation overhead. Manually redistributing the receptive field is difficult. We automate the process with neural architecture search to jointly optimize the neural architecture and inference scheduling, leading to MCUNetV2. Patch-based inference effectively reduces the peak memory usage of existing networks by 4-8x. Co-designed with neural networks, MCUNetV2 sets a record ImageNet accuracy on MCU (71.8%), and achieves >90% accuracy on the visual wake words dataset under only 32kB SRAM. MCUNetV2 also unblocks object detection on tiny devices, achieving 16.9% higher mAP on Pascal VOC compared to the state-of-the-art result. Our study largely addressed the memory bottleneck in tinyML and paved the way for various vision applications beyond image classification.

References in corpus (25)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• TensorFlow: A system for large-scale machine learning
• Neural Architecture Search with Reinforcement Learning
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• DARTS: Differentiable Architecture Search
• Searching for Activation Functions
• Trained Ternary Quantization
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Learning Efficient Convolutional Networks through Network Slimming
• MCUNet: Tiny Deep Learning on IoT Devices
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
• MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
• Visual Wake Words Dataset
• SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers
• Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets
• LFFD: A Light and Fast Face Detector for Edge Devices
• EXTD: Extremely Tiny Face Detector via Iterative Filter Reuse
• RNNPool: Efficient Non-linear Pooling for RAM Constrained Inference
• Optimally Scheduling CNN Convolutions for Efficient Memory Access
• Neural networks on microcontrollers: saving memory at inference via operator reordering
• Enabling Large Neural Networks on Tiny Microcontrollers with Swapping
• Ordering Chaos: Memory-Aware Scheduling of Irregularly Wired Neural Networks for Edge Devices

Cited by in corpus (11)

• Machine Learning for Microcontroller-Class Hardware: A Review
• Tiny Machine Learning: Progress and Futures
• Artificial Intelligence of Things: A Survey
• An Ultra-low Power TinyML System for Real-time Visual Processing at Edge
• TinyissimoYOLO: A Quantized, Low-Memory Footprint, TinyML Object Detection Network for Low Power Microcontrollers
• Flexible and Fully Quantized Ultra-Lightweight TinyissimoYOLO for Ultra-Low-Power Edge Systems
• HTVM: Efficient Neural Network Deployment On Heterogeneous TinyML Platforms
• Ultra-Efficient On-Device Object Detection on AI-Integrated Smart Glasses with TinyissimoYOLO
• Generic Neural Architecture Search via Regression
• ACCO: Automated Causal CNN Scheduling Optimizer for Real-Time Edge Accelerators
• Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression