Conformer: Convolution-augmented Transformer for Speech Recognition

Abstract

Recently Transformer and Convolution neural network (CNN) based models have shown promising results in Automatic Speech Recognition (ASR), outperforming Recurrent neural networks (RNNs). Transformer models are good at capturing content-based global interactions, while CNNs exploit local features effectively. In this work, we achieve the best of both worlds by studying how to combine convolution neural networks and transformers to model both local and global dependencies of an audio sequence in a parameter-efficient way. To this regard, we propose the convolution-augmented transformer for speech recognition, named Conformer. Conformer significantly outperforms the previous Transformer and CNN based models achieving state-of-the-art accuracies. On the widely used LibriSpeech benchmark, our model achieves WER of 2.1%/4.3% without using a language model and 1.9%/3.9% with an external language model on test/testother. We also observe competitive performance of 2.7%/6.3% with a small model of only 10M parameters.

Submitted to Interspeech 2020

References in corpus (9)

• Sequence Transduction with Recurrent Neural Networks
• Searching for Activation Functions
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
• ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context
• QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutions
• Convolutional Self-Attention Networks
• SpecAugment on Large Scale Datasets

Cited by in corpus (2)

• Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers
• On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition