Syntactically Supervised Transformers for Faster Neural Machine Translation

Abstract

Standard decoders for neural machine translation autoregressively generate a single target token per time step, which slows inference especially for long outputs. While architectural advances such as the Transformer fully parallelize the decoder computations at training time, inference still proceeds sequentially. Recent developments in non- and semi- autoregressive decoding produce multiple tokens per time step independently of the others, which improves inference speed but deteriorates translation quality. In this work, we propose the syntactically supervised Transformer (SynST), which first autoregressively predicts a chunked parse tree before generating all of the target tokens in one shot conditioned on the predicted parse. A series of controlled experiments demonstrates that SynST decodes sentences ~ 5x faster than the baseline autoregressive Transformer while achieving higher BLEU scores than most competing methods on En-De and En-Fr datasets.

arXiv comment9 pages, 5 figures, accepted to ACL 2019

References in corpus (7)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Non-Autoregressive Neural Machine Translation
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Syntactically Guided Neural Machine Translation
• Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoders
• Can Active Memory Replace Attention?

Cited by in corpus (20)

• HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
• Understanding Knowledge Distillation in Non-autoregressive Machine Translation
• Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information
• Hard-Coded Gaussian Attention for Neural Machine Translation
• Semantics-aware Attention Improves Neural Machine Translation
• Transformer-Based Neural Text Generation with Syntactic Guidance
• Neural Machine Translation: A Review of Methods, Resources, and Tools
• Encoding Syntactic Knowledge in Transformer Encoder for Intent Detection and Slot Filling
• Fast Interleaved Bidirectional Sequence Generation
• Non-Autoregressive Translation by Learning Target Categorical Codes
• Syntax-driven Iterative Expansion Language Models for Controllable Text Generation
• Sequence-Level Training for Non-Autoregressive Neural Machine Translation
• Exemplar-Controllable Paraphrasing and Translation using Bitext
• Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation
• Modeling Coverage for Non-Autoregressive Neural Machine Translation
• AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translate
• Data-to-text Generation by Splicing Together Nearest Neighbors
• Improved Latent Tree Induction with Distant Supervision via Span Constraints