XL-Editor: Post-editing Sentences with XLNet

Abstract

While neural sequence generation models achieve initial success for many NLP applications, the canonical decoding procedure with left-to-right generation order (i.e., autoregressive) in one-pass can not reflect the true nature of human revising a sentence to obtain a refined result. In this work, we propose XL-Editor, a novel training framework that enables state-of-the-art generalized autoregressive pretraining methods, XLNet specifically, to revise a given sentence by the variable-length insertion probability. Concretely, XL-Editor can (1) estimate the probability of inserting a variable-length sequence into a specific position of a given sentence; (2) execute post-editing operations such as insertion, deletion, and replacement based on the estimated variable-length insertion probability; (3) complement existing sequence-to-sequence models to refine the generated sequences. Empirically, we first demonstrate better post-editing capabilities of XL-Editor over XLNet on the text insertion and deletion tasks, which validates the effectiveness of our proposed framework. Furthermore, we extend XL-Editor to the unpaired text style transfer task, where transferring the target style onto a given sentence can be naturally viewed as post-editing the sentence into the target style. XL-Editor achieves significant improvement in style transfer accuracy and also maintains coherent semantic of the original sentence, showing the broad applicability of our method.

arXiv commentUnder review

References in corpus (10)

• Non-Autoregressive Neural Machine Translation
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• Style Transfer as Unsupervised Machine Translation
• KERMIT: Generative Insertion-Based Modeling for Sequences
• Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach
• A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models
• A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfer
• Iterative Refinement for Machine Translation
• Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation
• A Hierarchical Reinforced Sequence Operation Method for Unsupervised Text Style Transfer

Cited by in corpus (3)

• Enabling Language Models to Fill in the Blanks
• Narrative Incoherence Detection
• Towards a Universal NLG for Dialogue Systems and Simulators with Future Bridging