Streaming Transformer-based Acoustic Models Using Self-attention with Augmented Memory

Abstract

Transformer-based acoustic modeling has achieved great suc-cess for both hybrid and sequence-to-sequence speech recogni-tion. However, it requires access to the full sequence, and thecomputational cost grows quadratically with respect to the in-put sequence length. These factors limit its adoption for stream-ing applications. In this work, we proposed a novel augmentedmemory self-attention, which attends on a short segment of theinput sequence and a bank of memories. The memory bankstores the embedding information for all the processed seg-ments. On the librispeech benchmark, our proposed methodoutperforms all the existing streamable transformer methods bya large margin and achieved over 15% relative error reduction,compared with the widely used LC-BLSTM baseline. Our find-ings are also confirmed on some large internal datasets.

submitted to Interspeech 2020

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• A Comparative Study on Transformer vs RNN in Speech Applications
• Transformer-based Acoustic Modeling for Hybrid Speech Recognition
• Transformers with convolutional context for ASR
• Transformer-Transducer: End-to-End Speech Recognition with Self-Attention

Cited by in corpus (1)

• StreaMulT: Streaming Multimodal Transformer for Heterogeneous and Arbitrary Long Sequential Data