Enabling Large Neural Networks on Tiny Microcontrollers with Swapping

Abstract

Running neural networks (NNs) on microcontroller units (MCUs) is becoming increasingly important, but is very difficult due to the tiny SRAM size of MCU. Prior work proposes many algorithm-level techniques to reduce NN memory footprints, but all at the cost of sacrificing accuracy and generality, which disqualifies MCUs for many important use cases. We investigate a system solution for MCUs to execute NNs out of core: dynamically swapping NN data chunks between an MCU's tiny SRAM and its large, low-cost external flash. Out-of-core NNs on MCUs raise multiple concerns: execution slowdown, storage wear out, energy consumption, and data security. We present a study showing that none is a showstopper; the key benefit -- MCUs being able to run large NNs with full accuracy and generality -- triumphs the overheads. Our findings suggest that MCUs can play a much greater role in edge intelligence.

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Pruning Filters for Efficient ConvNets
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Hello Edge: Keyword Spotting on Microcontrollers
• MCUNet: Tiny Deep Learning on IoT Devices
• TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems
• IOS: Inter-Operator Scheduler for CNN Acceleration
• Out-of-core Training for Extremely Large-Scale Neural Networks With Adaptive Window-Based Scheduling

Cited by in corpus (4)

• Tiny Machine Learning: Progress and Futures
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
• SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget
• Power-Performance Characterization of TinyML Systems