TinyLSTMs: Efficient Neural Speech Enhancement for Hearing Aids

Abstract

Modern speech enhancement algorithms achieve remarkable noise suppression by means of large recurrent neural networks (RNNs). However, large RNNs limit practical deployment in hearing aid hardware (HW) form-factors, which are battery powered and run on resource-constrained microcontroller units (MCUs) with limited memory capacity and compute capability. In this work, we use model compression techniques to bridge this gap. We define the constraints imposed on the RNN by the HW and describe a method to satisfy them. Although model compression techniques are an active area of research, we are the first to demonstrate their efficacy for RNN speech enhancement, using pruning and integer quantization of weights/activations. We also demonstrate state update skipping, which reduces the computational load. Finally, we conduct a perceptual evaluation of the compressed models to verify audio quality on human raters. Results show a reduction in model size and operations of 11.9$\times$ and 2.9$\times$, respectively, over the baseline for compressed models, without a statistical difference in listening preference and only exhibiting a loss of 0.55dB SDR. Our model achieves a computational latency of 2.39ms, well within the 10ms target and 351$\times$ better than previous work.

First four authors contributed equally. For audio samples, see https://github.com/BoseCorp/efficient-neural-speech-enhancement

References in corpus (1)

• SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers

Cited by in corpus (19)

• Machine Learning for Microcontroller-Class Hardware: A Review
• MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers
• TinyML: Tools, Applications, Challenges, and Future Research Directions
• ClearBuds: Wireless Binaural Earbuds for Learning-Based Speech Enhancement
• TinyML for Ubiquitous Edge AI
• Inference skipping for more efficient real-time speech enhancement with parallel RNNs
• Real-time Denoising and Dereverberation with Tiny Recurrent U-Net
• Sparse Systolic Tensor Array for Efficient CNN Hardware Acceleration
• EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inference
• AnalogNets: ML-HW Co-Design of Noise-robust TinyML Models and Always-On Analog Compute-in-Memory Accelerator
• Dynamic nsNet2: Efficient Deep Noise Suppression with Early Exiting
• CCATMos: Convolutional Context-aware Transformer Network for Non-intrusive Speech Quality Assessment
• Principled Pruning of Bayesian Neural Networks through Variational Free Energy Minimization
• Time-domain Speech Enhancement with Generative Adversarial Learning
• Ultra-Lightweight Speech Separation via Group Communication
• Visual-Informed Speech Enhancement Using Attention-Based Beamforming
• Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices
• Adaptive Slimming for Scalable and Efficient Speech Enhancement
• Weight, Block or Unit? Exploring Sparsity Tradeoffs for Speech Enhancement on Tiny Neural Accelerators