Light Gated Recurrent Units for Speech Recognition

Abstract

A field that has directly benefited from the recent advances in deep learning is Automatic Speech Recognition (ASR). Despite the great achievements of the past decades, however, a natural and robust human-machine speech interaction still appears to be out of reach, especially in challenging environments characterized by significant noise and reverberation. To improve robustness, modern speech recognizers often employ acoustic models based on Recurrent Neural Networks (RNNs), that are naturally able to exploit large time contexts and long-term speech modulations. It is thus of great interest to continue the study of proper techniques for improving the effectiveness of RNNs in processing speech signals. In this paper, we revise one of the most popular RNN models, namely Gated Recurrent Units (GRUs), and propose a simplified architecture that turned out to be very effective for ASR. The contribution of this work is two-fold: First, we analyze the role played by the reset gate, showing that a significant redundancy with the update gate occurs. As a result, we propose to remove the former from the GRU design, leading to a more efficient and compact single-gate model. Second, we propose to replace hyperbolic tangent with ReLU activations. This variation couples well with batch normalization and could help the model learn long-term dependencies without numerical issues. Results show that the proposed architecture, called Light GRU (Li-GRU), not only reduces the per-epoch training time by more than 30% over a standard GRU, but also consistently improves the recognition accuracy across different tasks, input features, noisy conditions, as well as across different ASR paradigms, ranging from standard DNN-HMM speech recognizers to end-to-end CTC models.

arXiv commentCopyright 2018 IEEE

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Neural Machine Translation by Jointly Learning to Align and Translate
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• LSTM: A Search Space Odyssey
• On the difficulty of training Recurrent Neural Networks
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Spatial Diffuseness Features for DNN-Based Speech Recognition in Noisy and Reverberant Environments
• Minimal Gated Unit for Recurrent Neural Networks

Cited by in corpus (40)

• An Experimental Review on Deep Learning Architectures for Time Series Forecasting
• SpeechBrain: A General-Purpose Speech Toolkit
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• End-to-End Optimized Transmission over Dispersive Intensity-Modulated Channels Using Bidirectional Recurrent Neural Networks
• Experimental Review of Neural-based approaches for Network Intrusion Management
• Interpretable Convolutional Filters with SincNet
• SiTGRU: Single-Tunnelled Gated Recurrent Unit for Abnormality Detection
• DANSE: Data-driven Non-linear State Estimation of Model-free Process in Unsupervised Learning Setup
• A Comprehensive Survey of Machine Learning Applied to Radar Signal Processing
• Speech and Speaker Recognition from Raw Waveform with SincNet
• Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition?
• Speaker Distance Estimation in Enclosures from Single-Channel Audio
• Unsupervised Speech Recognition
• CSB-RNN: A Faster-than-Realtime RNN Acceleration Framework with Compressed Structured Blocks
• Delayed Memory Unit: Modelling Temporal Dependency Through Delay Gate
• Anisotropic Stroke Control for Multiple Artists Style Transfer
• Deep Neural Machine Translation with Weakly-Recurrent Units
• Automatic speech recognition for the Nepali language using CNN, bidirectional LSTM and ResNet
• Attention is All You Need in Speech Separation
• Effective Low-Cost Time-Domain Audio Separation Using Globally Attentive Locally Recurrent Networks
• Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps
• Learning Waveform-Based Acoustic Models using Deep Variational Convolutional Neural Networks
• Deep Spiking Neural Networks for Large Vocabulary Automatic Speech Recognition
• A Bayesian Approach to Recurrence in Neural Networks
• ClovaCall: Korean Goal-Oriented Dialog Speech Corpus for Automatic Speech Recognition of Contact Centers
• Conditional independence for pretext task selection in Self-supervised speech representation learning
• SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts
• Learning Speech Representations from Raw Audio by Joint Audiovisual Self-Supervision
• Normalizing Flow based Hidden Markov Models for Classification of Speech Phones with Explainability
• Particle Filter Recurrent Neural Networks
• Robust Classification using Hidden Markov Models and Mixtures of Normalizing Flows
• A Dynamically Controlled Recurrent Neural Network for Modeling Dynamical Systems
• A Spatio-temporal Prediction Methodology Based on Deep Learning and Real Wi-Fi Measurements
• Challenges and Opportunities of Speech Recognition for Bengali Language
• Gated recurrent units and temporal convolutional network for multilabel classification
• Learned Transferable Architectures Can Surpass Hand-Designed Architectures for Large Scale Speech Recognition
• Bayesian Recurrent Units and the Forward-Backward Algorithm
• CryptoGRU: Low Latency Privacy-Preserving Text Analysis With GRU
• Tune-In: Training Under Negative Environments with Interference for Attention Networks Simulating Cocktail Party Effect