Quantization and Deployment of Deep Neural Networks on Microcontrollers

Abstract

Embedding Artificial Intelligence onto low-power devices is a challenging task that has been partly overcome with recent advances in machine learning and hardware design. Presently, deep neural networks can be deployed on embedded targets to perform different tasks such as speech recognition,object detection or Human Activity Recognition. However, there is still room for optimization of deep neural networks onto embedded devices. These optimizations mainly address power consumption,memory and real-time constraints, but also an easier deployment at the edge. Moreover, there is still a need for a better understanding of what can be achieved for different use cases. This work focuses on quantization and deployment of deep neural networks onto low-power 32-bit microcontrollers. The quantization methods, relevant in the context of an embedded execution onto a microcontroller, are first outlined. Then, a new framework for end-to-end deep neural networks training, quantization and deployment is presented. This framework, called MicroAI, is designed as an alternative to existing inference engines (TensorFlow Lite for Microcontrollers and STM32CubeAI). Our framework can indeed be easily adjusted and/or extended for specific use cases. Execution using single precision 32-bit floating-point as well as fixed-point on 8- and 16-bit integers are supported. The proposed quantization method is evaluated with three different datasets (UCI-HAR, Spoken MNIST and GTSRB). Finally, a comparison study between MicroAI and both existing embedded inference engines is provided in terms of memory and power efficiency. On-device evaluation is done using ARM Cortex-M4F-based microcontrollers (Ambiq Apollo3 and STM32L452RE).

arXiv comment34 pages, 14 figures. Published in MDPI Sensors 2021, special issue "Embedded Artificial Intelligence (AI) for Smart Sensing and IoT Applications": https://www.mdpi.com/1424-8220/21/9/2984 . v2: add reference for MicroAI software and link to source code repository; fix Eq. 3 according to implementation; improve English grammar and spelling; improve page layout

References in corpus (16)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Mixed Precision Training
• Binarized Neural Networks
• Hello Edge: Keyword Spotting on Microcontrollers
• Training deep neural networks with low precision multiplications
• Learned Step Size Quantization
• MCUNet: Tiny Deep Learning on IoT Devices
• Benchmarking TPU, GPU, and CPU Platforms for Deep Learning
• TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems
• Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN)
• Iteratively Training Look-Up Tables for Network Quantization
• BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization

Cited by in corpus (8)

• Machine Learning for Microcontroller-Class Hardware: A Review
• Computer Vision Model Compression Techniques for Embedded Systems: A Survey
• Shifting Capsule Networks from the Cloud to the Deep Edge
• High-availability displacement sensing with multi-channel self mixing interferometry
• acoupi: An Open-Source Python Framework for Deploying Bioacoustic AI Models on Edge Devices
• Big-Little Adaptive Neural Networks on Low-Power Near-Subthreshold Processors
• Understanding the effect of sparsity on neural networks robustness
• OpTC -- A Toolchain for Deployment of Neural Networks on AURIX TC3xx Microcontrollers