Hello Edge: Keyword Spotting on Microcontrollers

Abstract

Keyword spotting (KWS) is a critical component for enabling speech based user interactions on smart devices. It requires real-time response and high accuracy for good user experience. Recently, neural networks have become an attractive choice for KWS architecture because of their superior accuracy compared to traditional speech processing algorithms. Due to its always-on nature, KWS application has highly constrained power budget and typically runs on tiny microcontrollers with limited memory and compute capability. The design of neural network architecture for KWS must consider these constraints. In this work, we perform neural network architecture evaluation and exploration for running KWS on resource-constrained microcontrollers. We train various neural network architectures for keyword spotting published in literature to compare their accuracy and memory/compute requirements. We show that it is possible to optimize these neural network architectures to fit within the memory and compute constraints of microcontrollers without sacrificing accuracy. We further explore the depthwise separable convolutional neural network (DS-CNN) and compare it against other neural network architectures. DS-CNN achieves an accuracy of 95.4%, which is ~10% higher than the DNN model with similar number of parameters.

Code available in github at https://github.com/ARM-software/ML-KWS-for-MCU

References in corpus (5)

• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Deep Convolutional Neural Network Inference with Floating-point Weights and Fixed-point Activations
• Dual Path Networks

Cited by in corpus (72)

• Federated Learning with Non-IID Data
• Machine Learning for Microcontroller-Class Hardware: A Review
• Quantization and Deployment of Deep Neural Networks on Microcontrollers
• Benchmarking TinyML Systems: Challenges and Direction
• TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems
• Keyword Transformer: A Self-Attention Model for Keyword Spotting
• MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers
• Edge Intelligence: Architectures, Challenges, and Applications
• A neural attention model for speech command recognition
• Intelligence at the Extreme Edge: A Survey on Reformable TinyML
• Streaming keyword spotting on mobile devices
• Visual Wake Words Dataset
• SkyNet: a Hardware-Efficient Method for Object Detection and Tracking on Embedded Systems
• Compact recurrent neural networks for acoustic event detection on low-energy low-complexity platforms
• A first look into the carbon footprint of federated learning
• ConvMixer: Feature Interactive Convolution with Curriculum Learning for Small Footprint and Noisy Far-field Keyword Spotting
• Learning Efficient Representations for Keyword Spotting with Triplet Loss
• TinyVers: A Tiny Versatile System-on-chip with State-Retentive eMRAM for ML Inference at the Extreme Edge
• A 23 $\mu$W Keyword Spotting IC with Ring-Oscillator-Based Time-Domain Feature Extraction
• Depthwise Separable Convolutional ResNet with Squeeze-and-Excitation Blocks for Small-footprint Keyword Spotting
• Federated Learning Meets Natural Language Processing: A Survey
• Temporal Convolution for Real-time Keyword Spotting on Mobile Devices
• Few-Shot Keyword Spotting in Any Language
• Few-Shot Keyword Spotting With Prototypical Networks
• Compressing RNNs for IoT devices by 15-38x using Kronecker Products
• Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollers
• SamurAI: A Versatile IoT Node With Event-Driven Wake-Up and Embedded ML Acceleration
• Run-Time Efficient RNN Compression for Inference on Edge Devices
• Enhancing Neural Architecture Search with Multiple Hardware Constraints for Deep Learning Model Deployment on Tiny IoT Devices
• Not All Ops Are Created Equal!
• Splitting Steepest Descent for Growing Neural Architectures
• PhyAug: Physics-Directed Data Augmentation for Deep Sensing Model Transfer in Cyber-Physical Systems
• FQ-Conv: Fully Quantized Convolution for Efficient and Accurate Inference
• Decentralizing Feature Extraction with Quantum Convolutional Neural Network for Automatic Speech Recognition
• Channel-wise Mixed-precision Assignment for DNN Inference on Constrained Edge Nodes
• Neural networks on microcontrollers: saving memory at inference via operator reordering
• Power-Performance Characterization of TinyML Systems
• Enabling Large Neural Networks on Tiny Microcontrollers with Swapping
• BiFSMN: Binary Neural Network for Keyword Spotting
• Edge Devices Inference Performance Comparison
• Gradient Scheduling with Global Momentum for Non-IID Data Distributed Asynchronous Training
• Rethinking Machine Learning Development and Deployment for Edge Devices
• Robustifying the Deployment of tinyML Models for Autonomous mini-vehicles
• An Investigation of Few-Shot Learning in Spoken Term Classification
• Small-footprint Keyword Spotting with Graph Convolutional Network
• Small-Footprint Open-Vocabulary Keyword Spotting with Quantized LSTM Networks
• Federated Learning for Keyword Spotting
• A Spike in Performance: Training Hybrid-Spiking Neural Networks with Quantized Activation Functions
• Discriminatory and orthogonal feature learning for noise robust keyword spotting
• Visual Keyword Spotting with Attention
• AutoKWS: Keyword Spotting with Differentiable Architecture Search
• Multi-Complexity-Loss DNAS for Energy-Efficient and Memory-Constrained Deep Neural Networks
• Low-Power Audio Keyword Spotting using Tsetlin Machines
• Behavior of Keyword Spotting Networks Under Noisy Conditions
• AUSN: Approximately Uniform Quantization by Adaptively Superimposing Non-uniform Distribution for Deep Neural Networks
• Neural ODE with Temporal Convolution and Time Delay Neural Networks for Small-Footprint Keyword Spotting
• A 14uJ/Decision Keyword Spotting Accelerator with In-SRAM-Computing and On Chip Learning for Customization
• Ultra-Low Power Keyword Spotting at the Edge
• JavaScript Convolutional Neural Networks for Keyword Spotting in the Browser: An Experimental Analysis
• Exploring Representation Learning for Small-Footprint Keyword Spotting
• A Lightweight dynamic filter for keyword spotting
• MLPerf Tiny Benchmark
• Improving Efficiency in Neural Network Accelerator Using Operands Hamming Distance optimization
• Small-Footprint Keyword Spotting with Multi-Scale Temporal Convolution
• Encoder-Decoder Neural Architecture Optimization for Keyword Spotting
• ACCO: Automated Causal CNN Scheduling Optimizer for Real-Time Edge Accelerators
• A Separable Temporal Convolution Neural Network with Attention for Small-Footprint Keyword Spotting
• On the human evaluation of audio adversarial examples
• Auto-KWS 2021 Challenge: Task, Datasets, and Baselines
• End-to-end Keyword Spotting using Xception-1d
• Spike-inspired Rank Coding for Fast and Accurate Recurrent Neural Networks
• Separable Temporal Convolution plus Temporally Pooled Attention for Lightweight High-performance Keyword Spotting