TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems

Abstract

Deep learning inference on embedded devices is a burgeoning field with myriad applications because tiny embedded devices are omnipresent. But we must overcome major challenges before we can benefit from this opportunity. Embedded processors are severely resource constrained. Their nearest mobile counterparts exhibit at least a 100 -- 1,000x difference in compute capability, memory availability, and power consumption. As a result, the machine-learning (ML) models and associated ML inference framework must not only execute efficiently but also operate in a few kilobytes of memory. Also, the embedded devices' ecosystem is heavily fragmented. To maximize efficiency, system vendors often omit many features that commonly appear in mainstream systems, including dynamic memory allocation and virtual memory, that allow for cross-platform interoperability. The hardware comes in many flavors (e.g., instruction-set architecture and FPU support, or lack thereof). We introduce TensorFlow Lite Micro (TF Micro), an open-source ML inference framework for running deep-learning models on embedded systems. TF Micro tackles the efficiency requirements imposed by embedded-system resource constraints and the fragmentation challenges that make cross-platform interoperability nearly impossible. The framework adopts a unique interpreter-based approach that provides flexibility while overcoming these challenges. This paper explains the design decisions behind TF Micro and describes its implementation details. Also, we present an evaluation to demonstrate its low resource requirement and minimal run-time performance overhead.

References in corpus (8)

• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Hello Edge: Keyword Spotting on Microcontrollers
• MCUNet: Tiny Deep Learning on IoT Devices
• Benchmarking TinyML Systems: Challenges and Direction
• Glow: Graph Lowering Compiler Techniques for Neural Networks
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
• Visual Wake Words Dataset
• A Cascade Architecture for Keyword Spotting on Mobile Devices

Cited by in corpus (20)

• Quantization and Deployment of Deep Neural Networks on Microcontrollers
• MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers
• Intelligence at the Extreme Edge: A Survey on Reformable TinyML
• Few-Shot Keyword Spotting in Any Language
• The Synergy of Complex Event Processing and Tiny Machine Learning in Industrial IoT
• MosquIoT: A System Based on IoT and Machine Learning for the Monitoring of Aedes aegypti (Diptera: Culicidae)
• Learning to Seek: Autonomous Source Seeking with Deep Reinforcement Learning Onboard a Nano Drone Microcontroller
• On-device Online Learning and Semantic Management of TinyML Systems
• Deep Learning-based ECG Classification on Raspberry PI using a Tensorflow Lite Model based on PTB-XL Dataset
• Software Engineering Approaches for TinyML based IoT Embedded Vision: A Systematic Literature Review
• TinyFedTL: Federated Transfer Learning on Tiny Devices
• Enabling Large Neural Networks on Tiny Microcontrollers with Swapping
• AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks
• MLonMCU: TinyML Benchmarking with Fast Retargeting
• Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for TinyML
• TinyOL: TinyML with Online-Learning on Microcontrollers
• MCUBERT: Memory-Efficient BERT Inference on Commodity Microcontrollers
• OpTC -- A Toolchain for Deployment of Neural Networks on AURIX TC3xx Microcontrollers
• MLPerf Tiny Benchmark
• On-device Federated Learning with Flower