Language Modeling with Deep Transformers

Abstract

We explore deep autoregressive Transformer models in language modeling for speech recognition. We focus on two aspects. First, we revisit Transformer model configurations specifically for language modeling. We show that well configured Transformer models outperform our baseline models based on the shallow stack of LSTM recurrent neural network layers. We carry out experiments on the open-source LibriSpeech 960hr task, for both 200K vocabulary word-level and 10K byte-pair encoding subword-level language modeling. We apply our word-level models to conventional hybrid speech recognition by lattice rescoring, and the subword-level models to attention based encoder-decoder models by shallow fusion. Second, we show that deep Transformer language models do not require positional encoding. The positional encoding is an essential augmentation for the self-attention mechanism which is invariant to sequence ordering. However, in autoregressive setup, as is the case for language modeling, the amount of information increases along the position dimension, which is a positional signal by its own. The analysis of attention weights shows that deep autoregressive self-attention models can automatically make use of such positional information. We find that removing the positional encoding even slightly improves the performance of these models.

To appear in the proceedings of INTERSPEECH 2019

References in corpus (15)

• TensorFlow: A system for large-scale machine learning
• LSTM: A Search Space Odyssey
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Gaussian Error Linear Units (GELUs)
• Convolutional Sequence to Sequence Learning
• A Structured Self-attentive Sentence Embedding
• Language Modeling with Gated Convolutional Networks
• On Using Monolingual Corpora in Neural Machine Translation
• Universal Transformers
• Improved training of end-to-end attention models for speech recognition
• A Comprehensive Study of Deep Bidirectional LSTM RNNs for Acoustic Modeling in Speech Recognition
• Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition
• Fully Convolutional Speech Recognition
• RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition
• On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition

Cited by in corpus (44)

• A Comparative Study on Transformer vs RNN in Speech Applications
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Intriguing Properties of Vision Transformers
• Code Prediction by Feeding Trees to Transformers
• Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss
• Centroid Transformers: Learning to Abstract with Attention
• A Simplified Fully Quantized Transformer for End-to-end Speech Recognition
• Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition
• Relative Positional Encoding for Transformers with Linear Complexity
• Efficient Training of Neural Transducer for Speech Recognition
• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition
• Generative Pre-Training for Speech with Autoregressive Predictive Coding
• An Empirical Study of Efficient ASR Rescoring with Transformers
• baller2vec: A Multi-Entity Transformer For Multi-Agent Spatiotemporal Modeling
• Long-span language modeling for speech recognition
• Contextualizing ASR Lattice Rescoring with Hybrid Pointer Network Language Model
• Adapting GPT, GPT-2 and BERT Language Models for Speech Recognition
• Cross-utterance Reranking Models with BERT and Graph Convolutional Networks for Conversational Speech Recognition
• Research on Modeling Units of Transformer Transducer for Mandarin Speech Recognition
• TensorCoder: Dimension-Wise Attention via Tensor Representation for Natural Language Modeling
• Transformer Language Models with LSTM-based Cross-utterance Information Representation
• End-to-end Audio-visual Speech Recognition with Conformers
• Advancing CTC-CRF Based End-to-End Speech Recognition with Wordpieces and Conformers
• RASR2: The RWTH ASR Toolkit for Generic Sequence-to-sequence Speech Recognition
• Private Language Model Adaptation for Speech Recognition
• The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization
• Bayesian Transformer Language Models for Speech Recognition
• Faster, Simpler and More Accurate Hybrid ASR Systems Using Wordpieces
• Combining TF-GridNet and Mixture Encoder for Continuous Speech Separation for Meeting Transcription
• On the Effectiveness of Neural Text Generation based Data Augmentation for Recognition of Morphologically Rich Speech
• Deep Transformer based Data Augmentation with Subword Units for Morphologically Rich Online ASR
• The RWTH ASR System for TED-LIUM Release 2: Improving Hybrid HMM with SpecAugment
• The DEformer: An Order-Agnostic Distribution Estimating Transformer
• TransfoRNN: Capturing the Sequential Information in Self-Attention Representations for Language Modeling
• Cross-Modal Transformer-Based Neural Correction Models for Automatic Speech Recognition
• Paradigm Shift in Language Modeling: Revisiting CNN for Modeling Sanskrit Originated Bengali and Hindi Language
• ASR Rescoring and Confidence Estimation with ELECTRA
• Leveraging Cross-Utterance Context For ASR Decoding
• Integrating Knowledge into End-to-End Speech Recognition from External Text-Only Data
• Multitask Learning and Joint Optimization for Transformer-RNN-Transducer Speech Recognition
• Neural Language Modeling With Implicit Cache Pointers
• A Parallelizable Lattice Rescoring Strategy with Neural Language Models
• On Sampling-Based Training Criteria for Neural Language Modeling