Benchmarking TinyML Systems: Challenges and Direction

Abstract

Recent advancements in ultra-low-power machine learning (TinyML) hardware promises to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted benchmark for these systems. Benchmarking allows us to measure and thereby systematically compare, evaluate, and improve the performance of systems and is therefore fundamental to a field reaching maturity. In this position paper, we present the current landscape of TinyML and discuss the challenges and direction towards developing a fair and useful hardware benchmark for TinyML workloads. Furthermore, we present our four benchmarks and discuss our selection methodology. Our viewpoints reflect the collective thoughts of the TinyMLPerf working group that is comprised of over 30 organizations.

arXiv comment6 pages, 1 figure, 3 tables

References in corpus (6)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Hello Edge: Keyword Spotting on Microcontrollers
• Description and Discussion on DCASE2020 Challenge Task2: Unsupervised Anomalous Sound Detection for Machine Condition Monitoring
• Visual Wake Words Dataset
• SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers
• MIMII Dataset: Sound Dataset for Malfunctioning Industrial Machine Investigation and Inspection

Cited by in corpus (26)

• TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems
• MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers
• Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights
• Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence
• A TinyML Platform for On-Device Continual Learning with Quantized Latent Replays
• Lightweight Neural Architecture Search for Temporal Convolutional Networks at the Edge
• CFU Playground: Full-Stack Open-Source Framework for Tiny Machine Learning (tinyML) Acceleration on FPGAs
• The Case for Retraining of ML Models for IoT Device Identification at the Edge
• Colab NAS: Obtaining lightweight task-specific convolutional neural networks following Occam's razor
• Software Engineering Approaches for TinyML based IoT Embedded Vision: A Systematic Literature Review
• Power-Performance Characterization of TinyML Systems
• TinyFedTL: Federated Transfer Learning on Tiny Devices
• Towards Performance Clarity of Edge Video Analytics
• Robustifying the Deployment of tinyML Models for Autonomous mini-vehicles
• Beyond Low Earth Orbit: Biomonitoring, Artificial Intelligence, and Precision Space Health
• Running hardware-aware neural architecture search on embedded devices under 512MB of RAM
• EdgeMark: An Automation and Benchmarking System for Embedded Artificial Intelligence Tools
• Towards large-scale, automated, accurate detection of CCTV camera objects using computer vision. Applications and implications for privacy, safety, and cybersecurity. (Preprint)
• U-TOE: Universal TinyML On-board Evaluation Toolkit for Low-Power IoT
• Multi-Complexity-Loss DNAS for Energy-Efficient and Memory-Constrained Deep Neural Networks
• StreamTinyNet: video streaming analysis with spatial-temporal TinyML
• Dendron: Enhancing Human Activity Recognition with On-Device TinyML Learning
• Machine Learning Systems for Intelligent Services in the IoT: A Survey
• Generalized Learning Vector Quantization for Classification in Randomized Neural Networks and Hyperdimensional Computing
• WDR FACE: The First Database for Studying Face Detection in Wide Dynamic Range
• Phoeni6: a Systematic Approach for Evaluating the Energy Consumption of Neural Networks