Streaming keyword spotting on mobile devices

Abstract

In this work we explore the latency and accuracy of keyword spotting (KWS) models in streaming and non-streaming modes on mobile phones. NN model conversion from non-streaming mode (model receives the whole input sequence and then returns the classification result) to streaming mode (model receives portion of the input sequence and classifies it incrementally) may require manual model rewriting. We address this by designing a Tensorflow/Keras based library which allows automatic conversion of non-streaming models to streaming ones with minimum effort. With this library we benchmark multiple KWS models in both streaming and non-streaming modes on mobile phones and demonstrate different tradeoffs between latency and accuracy. We also explore novel KWS models with multi-head attention which reduce the classification error over the state-of-art by 10% on Google speech commands data sets V2. The streaming library with all experiments is open-sourced.

References in corpus (10)

• Neural Machine Translation by Jointly Learning to Align and Translate
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Compressing Deep Convolutional Networks using Vector Quantization
• Hello Edge: Keyword Spotting on Microcontrollers
• A neural attention model for speech command recognition
• Temporal Convolution for Real-time Keyword Spotting on Mobile Devices
• MatchboxNet: 1D Time-Channel Separable Convolutional Neural Network Architecture for Speech Commands Recognition

Cited by in corpus (17)

• Keyword Transformer: A Self-Attention Model for Keyword Spotting
• Learning Efficient Representations for Keyword Spotting with Triplet Loss
• ConvMixer: Feature Interactive Convolution with Curriculum Learning for Small Footprint and Noisy Far-field Keyword Spotting
• A 23 $\mu$W Keyword Spotting IC with Ring-Oscillator-Based Time-Domain Feature Extraction
• AST: Audio Spectrogram Transformer
• Broadcasted Residual Learning for Efficient Keyword Spotting
• Hardware Aware Training for Efficient Keyword Spotting on General Purpose and Specialized Hardware
• AutoKWS: Keyword Spotting with Differentiable Architecture Search
• Self-Learning for Personalized Keyword Spotting on Ultra-Low-Power Audio Sensors
• Effective Integration of KAN for Keyword Spotting
• A Lightweight dynamic filter for keyword spotting
• Small Footprint Convolutional Recurrent Networks for Streaming Wakeword Detection
• Real-time Speech Frequency Bandwidth Extension
• GE2E-KWS: Generalized End-to-End Training and Evaluation for Zero-shot Keyword Spotting
• Encoder-Decoder Neural Architecture Optimization for Keyword Spotting
• Audiomer: A Convolutional Transformer For Keyword Spotting
• Attention-Free Keyword Spotting