A Probabilistic Formulation of Unsupervised Text Style Transfer

Abstract

We present a deep generative model for unsupervised text style transfer that unifies previously proposed non-generative techniques. Our probabilistic approach models non-parallel data from two domains as a partially observed parallel corpus. By hypothesizing a parallel latent sequence that generates each observed sequence, our model learns to transform sequences from one domain to another in a completely unsupervised fashion. In contrast with traditional generative sequence models (e.g. the HMM), our model makes few assumptions about the data it generates: it uses a recurrent language model as a prior and an encoder-decoder as a transduction distribution. While computation of marginal data likelihood is intractable in this model class, we show that amortized variational inference admits a practical surrogate. Further, by drawing connections between our variational objective and other recent unsupervised style transfer and machine translation techniques, we show how our probabilistic view can unify some known non-generative objectives such as backtranslation and adversarial loss. Finally, we demonstrate the effectiveness of our method on a wide range of unsupervised style transfer tasks, including sentiment transfer, formality transfer, word decipherment, author imitation, and related language translation. Across all style transfer tasks, our approach yields substantial gains over state-of-the-art non-generative baselines, including the state-of-the-art unsupervised machine translation techniques that our approach generalizes. Further, we conduct experiments on a standard unsupervised machine translation task and find that our unified approach matches the current state-of-the-art.

ICLR 2020 conference paper (spotlight). The first two authors contributed equally

References in corpus (5)

• Auto-Encoding Variational Bayes
• Categorical Reparameterization with Gumbel-Softmax
• Style Transfer from Non-Parallel Text by Cross-Alignment
• Dual Learning for Machine Translation
• An Effective Approach to Unsupervised Machine Translation

Cited by in corpus (19)

• From Theories on Styles to their Transfer in Text: Bridging the Gap with a Hierarchical Survey
• Cycle-Consistent Adversarial Autoencoders for Unsupervised Text Style Transfer
• Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer
• Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer
• On Learning Text Style Transfer with Direct Rewards
• Neural Data-to-Text Generation with LM-based Text Augmentation
• Enhancing Content Preservation in Text Style Transfer Using Reverse Attention and Conditional Layer Normalization
• LEWIS: Levenshtein Editing for Unsupervised Text Style Transfer
• Dual Reconstruction: a Unifying Objective for Semi-Supervised Neural Machine Translation
• TextSETTR: Few-Shot Text Style Extraction and Tunable Targeted Restyling
• Inference Time Style Control for Summarization
• NAST: A Non-Autoregressive Generator with Word Alignment for Unsupervised Text Style Transfer
• Preventing Author Profiling through Zero-Shot Multilingual Back-Translation
• Finetuning Pretrained Transformers into Variational Autoencoders
• Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer
• Transferable Persona-Grounded Dialogues via Grounded Minimal Edits
• Transductive Learning for Unsupervised Text Style Transfer
• Syntax Matters! Syntax-Controlled in Text Style Transfer
• Style Pooling: Automatic Text Style Obfuscation for Improved Classification Fairness