Multi-encoder multi-resolution framework for end-to-end speech recognition

Abstract

Attention-based methods and Connectionist Temporal Classification (CTC) network have been promising research directions for end-to-end Automatic Speech Recognition (ASR). The joint CTC/Attention model has achieved great success by utilizing both architectures during multi-task training and joint decoding. In this work, we present a novel Multi-Encoder Multi-Resolution (MEMR) framework based on the joint CTC/Attention model. Two heterogeneous encoders with different architectures, temporal resolutions and separate CTC networks work in parallel to extract complimentary acoustic information. A hierarchical attention mechanism is then used to combine the encoder-level information. To demonstrate the effectiveness of the proposed model, experiments are conducted on Wall Street Journal (WSJ) and CHiME-4, resulting in relative Word Error Rate (WER) reduction of 18.0-32.1%. Moreover, the proposed MEMR model achieves 3.6% WER in the WSJ eval92 test set, which is the best WER reported for an end-to-end system on this benchmark.

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding
• Residual Convolutional CTC Networks for Automatic Speech Recognition
• Towards better decoding and language model integration in sequence to sequence models
• Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM
• Improved Regularization Techniques for End-to-End Speech Recognition
• End-to-end Speech Recognition with Word-based RNN Language Models

Cited by in corpus (4)

• Stream attention-based multi-array end-to-end speech recognition
• Multi-QuartzNet: Multi-Resolution Convolution for Speech Recognition with Multi-Layer Feature Fusion
• Performance Monitoring for End-to-End Speech Recognition
• A practical two-stage training strategy for multi-stream end-to-end speech recognition