Neural networks on microcontrollers: saving memory at inference via operator reordering

Abstract

Designing deep learning models for highly-constrained hardware would allow imbuing many edge devices with intelligence. Microcontrollers (MCUs) are an attractive platform for building smart devices due to their low cost, wide availability, and modest power usage. However, they lack the computational resources to run neural networks as straightforwardly as mobile or server platforms, which necessitates changes to the network architecture and the inference software. In this work, we discuss the deployment and memory concerns of neural networks on MCUs and present a way of saving memory by changing the execution order of the network's operators, which is orthogonal to other compression methods. We publish a tool for reordering operators of TensorFlow Lite models and demonstrate its utility by sufficiently reducing the memory footprint of a CNN to deploy it on an MCU with 512KB SRAM.

The tool is available at https://github.com/oxmlsys/tflite-tools

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Hello Edge: Keyword Spotting on Microcontrollers
• Faster gaze prediction with dense networks and Fisher pruning
• Visual Wake Words Dataset
• SwiftNet: Using Graph Propagation as Meta-knowledge to Search Highly Representative Neural Architectures

Cited by in corpus (5)

• MCUNet: Tiny Deep Learning on IoT Devices
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
• Searching for Winograd-aware Quantized Networks
• Efficient Neural Network Deployment for Microcontroller
• The Final Frontier: Deep Learning in Space