A Neural Transducer

Abstract

Sequence-to-sequence models have achieved impressive results on various tasks. However, they are unsuitable for tasks that require incremental predictions to be made as more data arrives or tasks that have long input sequences and output sequences. This is because they generate an output sequence conditioned on an entire input sequence. In this paper, we present a Neural Transducer that can make incremental predictions as more input arrives, without redoing the entire computation. Unlike sequence-to-sequence models, the Neural Transducer computes the next-step distribution conditioned on the partially observed input sequence and the partially generated sequence. At each time step, the transducer can decide to emit zero to many output symbols. The data can be processed using an encoder and presented as input to the transducer. The discrete decision to emit a symbol at every time step makes it difficult to learn with conventional backpropagation. It is however possible to train the transducer by using a dynamic programming algorithm to generate target discrete decisions. Our experiments show that the Neural Transducer works well in settings where it is required to produce output predictions as data come in. We also find that the Neural Transducer performs well for long sequences even when attention mechanisms are not used.

References in corpus (8)

• Sequence to Sequence Learning with Neural Networks
• Recurrent Neural Network Regularization
• Sequence Transduction with Recurrent Neural Networks
• End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
• Neural Programmer-Interpreters
• Show and Tell: A Neural Image Caption Generator
• A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
• Neural Programmer: Inducing Latent Programs with Gradient Descent

Cited by in corpus (20)

• Recent Advances in Recurrent Neural Networks
• Online and Linear-Time Attention by Enforcing Monotonic Alignments
• Monotonic Chunkwise Attention
• Towards Online End-to-end Transformer Automatic Speech Recognition
• Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss
• Joint Online Spoken Language Understanding and Language Modeling with Recurrent Neural Networks
• A Comparison of Label-Synchronous and Frame-Synchronous End-to-End Models for Speech Recognition
• Learning Online Alignments with Continuous Rewards Policy Gradient
• Incremental Text to Speech for Neural Sequence-to-Sequence Models using Reinforcement Learning
• Online Segment to Segment Neural Transduction
• Gated Recurrent Context: Softmax-free Attention for Online Encoder-Decoder Speech Recognition
• Morphological Inflection Generation with Hard Monotonic Attention
• Streaming Object Detection for 3-D Point Clouds
• One In A Hundred: Select The Best Predicted Sequence from Numerous Candidates for Streaming Speech Recognition
• Synchronous Transformers for End-to-End Speech Recognition
• Minimum Latency Training Strategies for Streaming Sequence-to-Sequence ASR
• Transformer ASR with Contextual Block Processing
• Attention based on-device streaming speech recognition with large speech corpus
• Multi-mode Transformer Transducer with Stochastic Future Context
• FSR: Accelerating the Inference Process of Transducer-Based Models by Applying Fast-Skip Regularization