Streaming Transformer ASR with Blockwise Synchronous Beam Search

Abstract

The Transformer self-attention network has shown promising performance as an alternative to recurrent neural networks in end-to-end (E2E) automatic speech recognition (ASR) systems. However, Transformer has a drawback in that the entire input sequence is required to compute both self-attention and source--target attention. In this paper, we propose a novel blockwise synchronous beam search algorithm based on blockwise processing of encoder to perform streaming E2E Transformer ASR. In the beam search, encoded feature blocks are synchronously aligned using a block boundary detection technique, where a reliability score of each predicted hypothesis is evaluated based on the end-of-sequence and repeated tokens in the hypothesis. Evaluations of the HKUST and AISHELL-1 Mandarin, LibriSpeech English, and CSJ Japanese tasks show that the proposed streaming Transformer algorithm outperforms conventional online approaches, including monotonic chunkwise attention (MoChA), especially when using the knowledge distillation technique. An ablation study indicates that our streaming approach contributes to reducing the response time, and the repetition criterion contributes significantly in certain tasks. Our streaming ASR models achieve comparable or superior performance to batch models and other streaming-based Transformer methods in all tasks considered.

arXiv commentAccepted for SLT 2021

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Attention-Based Models for Speech Recognition
• A Comparative Study on Transformer vs RNN in Speech Applications
• Generating Long Sequences with Sparse Transformers
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition
• Self-Attention Transducers for End-to-End Speech Recognition
• ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context
• Towards Online End-to-end Transformer Automatic Speech Recognition