Accelerating Transformer Inference for Translation via Parallel Decoding

Abstract

Autoregressive decoding limits the efficiency of transformers for Machine Translation (MT). The community proposed specific network architectures and learning-based methods to solve this issue, which are expensive and require changes to the MT model, trading inference speed at the cost of the translation quality. In this paper, we propose to address the problem from the point of view of decoding algorithms, as a less explored but rather compelling direction. We propose to reframe the standard greedy autoregressive decoding of MT with a parallel formulation leveraging Jacobi and Gauss-Seidel fixed-point iteration methods for fast inference. This formulation allows to speed up existing models without training or modifications while retaining translation quality. We present three parallel decoding algorithms and test them on different languages and models showing how the parallelization introduces a speedup up to 38% w.r.t. the standard autoregressive decoding and nearly 2x when scaling the method on parallel resources. Finally, we introduce a decoding dependency graph visualizer (DDGviz) that let us see how the model has learned the conditional dependence between tokens and inspect the decoding procedure.

Accepted at ACL 2023 main conference

References in corpus (10)

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Aligned Cross Entropy for Non-Autoregressive Machine Translation
• Semi-Autoregressive Training Improves Mask-Predict Decoding
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translation
• Fast Inference from Transformers via Speculative Decoding
• Step-unrolled Denoising Autoencoders for Text Generation
• Accelerating Large Language Model Decoding with Speculative Sampling
• Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervision

Cited by in corpus (3)

• Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
• Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation
• Confidence-Modulated Speculative Decoding for Large Language Models