Big Bidirectional Insertion Representations for Documents

Abstract

The Insertion Transformer is well suited for long form text generation due to its parallel generation capabilities, requiring $O({\log }_{2}n)$ generation steps to generate $n$ tokens. However, modeling long sequences is difficult, as there is more ambiguity captured in the attention mechanism. This work proposes the Big Bidirectional Insertion Representations for Documents (Big BIRD), an insertion-based model for document-level translation tasks. We scale up the insertion-based models to long form documents. Our key contribution is introducing sentence alignment via sentence-positional embeddings between the source and target document. We show an improvement of +4.3 BLEU on the WMT'19 English$\to$German document-level translation task compared with the Insertion Transformer baseline.

References in corpus (6)

• Sequence to Sequence Learning with Neural Networks
• Achieving Human Parity on Automatic Chinese to English News Translation
• Insertion Transformer: Flexible Sequence Generation via Insertion Operations
• KERMIT: Generative Insertion-Based Modeling for Sequences
• Non-Monotonic Sequential Text Generation
• Insertion-based Decoding with automatically Inferred Generation Order