Transformers with convolutional context for ASR

Abstract

The recent success of transformer networks for neural machine translation and other NLP tasks has led to a surge in research work trying to apply it for speech recognition. Recent efforts studied key research questions around ways of combining positional embedding with speech features, and stability of optimization for large scale learning of transformer networks. In this paper, we propose replacing the sinusoidal positional embedding for transformers with convolutionally learned input representations. These contextual representations provide subsequent transformer blocks with relative positional information needed for discovering long-range relationships between local concepts. The proposed system has favorable optimization characteristics where our reported results are produced with fixed learning rate of 1.0 and no warmup steps. The proposed model achieves a competitive 4.7% and 12.9% WER on the Librispeech ``test clean'' and ``test other'' subsets when no extra LM text is provided.

References in corpus (8)

• Sequence to Sequence Learning with Neural Networks
• ADADELTA: An Adaptive Learning Rate Method
• Attention-Based Models for Speech Recognition
• Improved training of end-to-end attention models for speech recognition
• Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
• Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition
• Fully Convolutional Speech Recognition
• The CAPIO 2017 Conversational Speech Recognition System

Cited by in corpus (49)

• Linformer: Self-Attention with Linear Complexity
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Transformer-based Acoustic Modeling for Hybrid Speech Recognition
• End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
• Random Feature Attention
• Effectiveness of self-supervised pre-training for speech recognition
• fairseq S2T: Fast Speech-to-Text Modeling with fairseq
• Transformer-Transducer: End-to-End Speech Recognition with Self-Attention
• DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization
• Uncertainty Estimation in Autoregressive Structured Prediction
• Iterative Pseudo-Labeling for Speech Recognition
• Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss
• Semantic Mask for Transformer based End-to-End Speech Recognition
• Imputer: Sequence Modelling via Imputation and Dynamic Programming
• A Simplified Fully Quantized Transformer for End-to-end Speech Recognition
• Espresso: A Fast End-to-end Neural Speech Recognition Toolkit
• Relative Positional Encoding for Transformers with Linear Complexity
• SparseBERT: Rethinking the Importance Analysis in Self-attention
• Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition
• Enforcing Encoder-Decoder Modularity in Sequence-to-Sequence Models
• Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translation
• Generative Pre-Training for Speech with Autoregressive Predictive Coding
• Transformer-XL Based Music Generation with Multiple Sequences of Time-valued Notes
• Neural Machine Translation with Byte-Level Subwords
• XLST: Cross-lingual Self-training to Learn Multilingual Representation for Low Resource Speech Recognition
• CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings
• Multiresolution and Multimodal Speech Recognition with Transformers
• Darts-Conformer: Towards Efficient Gradient-Based Neural Architecture Search For End-to-End ASR
• MixSpeech: Data Augmentation for Low-resource Automatic Speech Recognition
• Visual Keyword Spotting with Attention
• CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition
• Multitask Learning for Low Resource Spoken Language Understanding
• Masked Pre-trained Encoder base on Joint CTC-Transformer
• Weak-Attention Suppression For Transformer Based Speech Recognition
• Improving RNN transducer with normalized jointer network
• Self-Training for End-to-End Speech Translation
• Transformer with Bidirectional Decoder for Speech Recognition
• Transformer Network for Semantically-Aware and Speech-Driven Upper-Face Generation
• Transformer-based end-to-end speech recognition with residual Gaussian-based self-attention
• Towards Better Modeling Hierarchical Structure for Self-Attention with Ordered Neurons
• TransMask: A Compact and Fast Speech Separation Model Based on Transformer
• Streaming Transformer-based Acoustic Models Using Self-attention with Augmented Memory
• Nana-HDR: A Non-attentive Non-autoregressive Hybrid Model for TTS
• Multitask Learning and Joint Optimization for Transformer-RNN-Transducer Speech Recognition
• Mutually-Constrained Monotonic Multihead Attention for Online ASR
• Gaussian Kernelized Self-Attention for Long Sequence Data and Its Application to CTC-based Speech Recognition
• Poly-NL: Linear Complexity Non-local Layers with Polynomials
• CarneliNet: Neural Mixture Model for Automatic Speech Recognition
• Improving Streaming Transformer Based ASR Under a Framework of Self-supervised Learning