Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring

Abstract

This article describes an efficient end-to-end speech translation (E2E-ST) framework based on non-autoregressive (NAR) models. End-to-end speech translation models have several advantages over traditional cascade systems such as inference latency reduction. However, conventional AR decoding methods are not fast enough because each token is generated incrementally. NAR models, however, can accelerate the decoding speed by generating multiple tokens in parallel on the basis of the token-wise conditional independence assumption. We propose a unified NAR E2E-ST framework called Orthros, which has an NAR decoder and an auxiliary shallow AR decoder on top of the shared encoder. The auxiliary shallow AR decoder selects the best hypothesis by rescoring multiple candidates generated from the NAR decoder in parallel (parallel AR rescoring). We adopt conditional masked language model (CMLM) and a connectionist temporal classification (CTC)-based model as NAR decoders for Orthros, referred to as Orthros-CMLM and Orthros-CTC, respectively. We also propose two training methods to enhance the CMLM decoder. Experimental evaluations on three benchmark datasets with six language directions demonstrated that Orthros achieved large improvements in translation quality with a very small overhead compared with the baseline NAR model. Moreover, the Conformer encoder architecture enabled large quality improvements, especially for CTC-based models. Orthros-CTC with the Conformer encoder increased decoding speed by 3.63x on CPU with translation quality comparable to that of an AR model.

References in corpus (23)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• A Comparative Study on Transformer vs RNN in Speech Applications
• FastSpeech: Fast, Robust and Controllable Text to Speech
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Insertion Transformer: Flexible Sequence Generation via Insertion Operations
• Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
• Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation
• fairseq S2T: Fast Speech-to-Text Modeling with fairseq
• KERMIT: Generative Insertion-Based Modeling for Sequences
• Semi-Autoregressive Training Improves Mask-Predict Decoding
• Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translation
• Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
• Bridging the Modality Gap for Speech-to-Text Translation
• Understanding Knowledge Distillation in Non-autoregressive Machine Translation
• Imitation Learning for Non-Autoregressive Neural Machine Translation
• Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
• An EM Approach to Non-autoregressive Conditional Sequence Generation
• Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct Speech Translation
• "Listen, Understand and Translate": Triple Supervision Decouples End-to-end Speech-to-text Translation
• Consecutive Decoding for Speech-to-text Translation
• Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders
• Hybrid-Regressive Neural Machine Translation

Cited by in corpus (1)

• Does Simultaneous Speech Translation need Simultaneous Models?