Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediates

Abstract

The multi-decoder (MD) end-to-end speech translation model has demonstrated high translation quality by searching for better intermediate automatic speech recognition (ASR) decoder states as hidden intermediates (HI). It is a two-pass decoding model decomposing the overall task into ASR and machine translation sub-tasks. However, the decoding speed is not fast enough for real-world applications because it conducts beam search for both sub-tasks during inference. We propose Fast-MD, a fast MD model that generates HI by non-autoregressive (NAR) decoding based on connectionist temporal classification (CTC) outputs followed by an ASR decoder. We investigated two types of NAR HI: (1) parallel HI by using an autoregressive Transformer ASR decoder and (2) masked HI by using Mask-CTC, which combines CTC and the conditional masked language model. To reduce a mismatch in the ASR decoder between teacher-forcing during training and conditioning on CTC outputs during testing, we also propose sampling CTC outputs during training. Experimental evaluations on three corpora show that Fast-MD achieved about 2x and 4x faster decoding speed than that of the naïve MD model on GPU and CPU with comparable translation quality. Adopting the Conformer encoder and intermediate CTC loss further boosts its quality without sacrificing decoding speed.

arXiv commentAccepted at IEEE ASRU 2021

References in corpus (13)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• A Comparative Study on Transformer vs RNN in Speech Applications
• fairseq S2T: Fast Speech-to-Text Modeling with fairseq
• Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition
• Bridging the Modality Gap for Speech-to-Text Translation
• Pushing the Limits of Non-Autoregressive Speech Recognition
• Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct Speech Translation
• "Listen, Understand and Translate": Triple Supervision Decouples End-to-end Speech-to-text Translation
• Consecutive Decoding for Speech-to-text Translation
• Enforcing Encoder-Decoder Modularity in Sequence-to-Sequence Models
• WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognition
• One In A Hundred: Select The Best Predicted Sequence from Numerous Candidates for Streaming Speech Recognition
• Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translation