End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results

Abstract

We replace the Hidden Markov Model (HMM) which is traditionally used in in continuous speech recognition with a bi-directional recurrent neural network encoder coupled to a recurrent neural network decoder that directly emits a stream of phonemes. The alignment between the input and output sequences is established using an attention mechanism: the decoder emits each symbol based on a context created with a subset of input symbols elected by the attention mechanism. We report initial results demonstrating that this new approach achieves phoneme error rates that are comparable to the state-of-the-art HMM-based decoders, on the TIMIT dataset.

As accepted to: Deep Learning and Representation Learning Workshop, NIPS 2014

References in corpus (7)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Improving neural networks by preventing co-adaptation of feature detectors
• ADADELTA: An Adaptive Learning Rate Method
• On the difficulty of training Recurrent Neural Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Sequence Transduction with Recurrent Neural Networks
• Pylearn2: a machine learning research library

Cited by in corpus (115)

• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Attention-Based Models for Speech Recognition
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Neural NILM: Deep Neural Networks Applied to Energy Disaggregation
• Deep Audio-Visual Speech Recognition
• Attention in Natural Language Processing
• Grammar as a Foreign Language
• Deep Learning for Time Series Forecasting: Tutorial and Literature Survey
• Improved training of end-to-end attention models for speech recognition
• GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs
• ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
• EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding
• Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networks
• Reinforcement Learning Neural Turing Machines - Revised
• A Survey of Deep Learning Techniques for Neural Machine Translation
• GRAM: Graph-based Attention Model for Healthcare Representation Learning
• AttSum: Joint Learning of Focusing and Summarization with Neural Attention
• Reward Augmented Maximum Likelihood for Neural Structured Prediction
• ESPnet: End-to-End Speech Processing Toolkit
• ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs
• Temporal Attention Model for Neural Machine Translation
• Local Monotonic Attention Mechanism for End-to-End Speech and Language Processing
• Joint Line Segmentation and Transcription for End-to-End Handwritten Paragraph Recognition
• Multichannel End-to-end Speech Recognition
• Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition
• Res3ATN -- Deep 3D Residual Attention Network for Hand Gesture Recognition in Videos
• Citrinet: Closing the Gap between Non-Autoregressive and Autoregressive End-to-End Models for Automatic Speech Recognition
• Implicit Distortion and Fertility Models for Attention-based Encoder-Decoder NMT Model
• Computer-assisted Pronunciation Training -- Speech synthesis is almost all you need
• Modeling Time Series Similarity with Siamese Recurrent Networks
• A Neural Transducer
• Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems
• An Attention-based Collaboration Framework for Multi-View Network Representation Learning
• Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling
• Improving Generalization of Transformer for Speech Recognition with Parallel Schedule Sampling and Relative Positional Embedding
• U2++: Unified Two-pass Bidirectional End-to-end Model for Speech Recognition
• Predicting Aircraft Trajectories: A Deep Generative Convolutional Recurrent Neural Networks Approach
• Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM
• Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNet
• Sequential Context Encoding for Duplicate Removal
• Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning
• Maneuver-Aware Pooling for Vehicle Trajectory Prediction
• Recursive Social Behavior Graph for Trajectory Prediction
• Listening while Speaking: Speech Chain by Deep Learning
• WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit
• A Multitask Deep Learning Approach for User Depression Detection on Sina Weibo
• Espresso: A Fast End-to-end Neural Speech Recognition Toolkit
• A GRU-based Encoder-Decoder Approach with Attention for Online Handwritten Mathematical Expression Recognition
• Improving the Performance of Neural Machine Translation Involving Morphologically Rich Languages
• LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition
• Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition
• Attention-based Wav2Text with Feature Transfer Learning
• Automatic Spelling Correction with Transformer for CTC-based End-to-End Speech Recognition
• WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognition
• Few-shot learning with attention-based sequence-to-sequence models
• Recurrent Models for Auditory Attention in Multi-Microphone Distance Speech Recognition
• End2End Acoustic to Semantic Transduction
• End-to-End ASR for Code-switched Hindi-English Speech
• Exploring RNN-Transducer for Chinese Speech Recognition
• An Online Attention-based Model for Speech Recognition
• Towards Language-Universal End-to-End Speech Recognition
• Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networks
• Improving End-To-End Modeling for Mispronunciation Detection with Effective Augmentation Mechanisms
• Joint Modeling of Accents and Acoustics for Multi-Accent Speech Recognition
• Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakers
• Multi-Modal Data Augmentation for End-to-End ASR
• Teaching Machines to Converse
• Maneuver-based Anchor Trajectory Hypotheses at Roundabouts
• Advancing CTC-CRF Based End-to-End Speech Recognition with Wordpieces and Conformers
• OkwuGbé: End-to-End Speech Recognition for Fon and Igbo
• GraphTCN: Spatio-Temporal Interaction Modeling for Human Trajectory Prediction
• Cross-Attention End-to-End ASR for Two-Party Conversations
• Unsupervised Speaker Adaptation using Attention-based Speaker Memory for End-to-End ASR
• Parsing Speech: A Neural Approach to Integrating Lexical and Acoustic-Prosodic Information
• Read, Tag, and Parse All at Once, or Fully-neural Dependency Parsing
• Auxiliary Interference Speaker Loss for Target-Speaker Speech Recognition
• End-to-end attention-based distant speech recognition with Highway LSTM
• End-to-End Monaural Multi-speaker ASR System without Pretraining
• End to End ASR System with Automatic Punctuation Insertion
• Learning Monocular Visual Odometry via Self-Supervised Long-Term Modeling
• Efficient Neural Architecture Search for End-to-end Speech Recognition via Straight-Through Gradients
• Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphone
• Self-and-Mixed Attention Decoder with Deep Acoustic Structure for Transformer-based LVCSR
• Promising Accurate Prefix Boosting for sequence-to-sequence ASR
• Language model integration based on memory control for sequence to sequence speech recognition
• CNN-based MultiChannel End-to-End Speech Recognition for everyday home environments
• End-to-End Automatic Speech Recognition Integrated With CTC-Based Voice Activity Detection
• Tackling Sequence to Sequence Mapping Problems with Neural Networks
• Dialog-context aware end-to-end speech recognition
• Tatum-Level Drum Transcription Based on a Convolutional Recurrent Neural Network with Language Model-Based Regularized Training
• KoSpeech: Open-Source Toolkit for End-to-End Korean Speech Recognition
• Attention-Based End-to-End Speech Recognition on Voice Search
• Neural Machine Translation: A Review and Survey
• On the Learning Dynamics of Attention Networks
• CAT: A CTC-CRF based ASR Toolkit Bridging the Hybrid and the End-to-end Approaches towards Data Efficiency and Low Latency
• Speech Corpus of Ainu Folklore and End-to-end Speech Recognition for Ainu Language
• Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models
• Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systems
• End-to-end speech-to-dialog-act recognition
• Modality Attention for End-to-End Audio-visual Speech Recognition
• Multi-Head Decoder for End-to-End Speech Recognition
• End-to-End Language Identification using Multi-Head Self-Attention and 1D Convolutional Neural Networks
• Joint Language Identification of Code-Switching Speech using Attention based E2E Network
• On Multilingual Training of Neural Dependency Parsers
• Neural Sequence Model Training via $\alpha$-divergence Minimization
• Low-Resource Contextual Topic Identification on Speech
• Approaches to Improving Recognition of Underrepresented Named Entities in Hybrid ASR Systems
• Generative Adversarial Training Data Adaptation for Very Low-resource Automatic Speech Recognition
• Where are we in semantic concept extraction for Spoken Language Understanding?
• End-to-End Code Switching Language Models for Automatic Speech Recognition
• Incorporating Language Level Information into Acoustic Models
• End-to-end feature fusion siamese network for adaptive visual tracking
• Probability Trajectory: One New Movement Description for Trajectory Prediction
• Acoustic-to-Word Models with Conversational Context Information
• Text-Only Domain Adaptation for End-to-End Speech Recognition through Down-Sampling Acoustic Representation