Educating Text Autoencoders: Latent Representation Guidance via Denoising

Abstract

Generative autoencoders offer a promising approach for controllable text generation by leveraging their latent sentence representations. However, current models struggle to maintain coherent latent spaces required to perform meaningful text manipulations via latent vector operations. Specifically, we demonstrate by example that neural encoders do not necessarily map similar sentences to nearby latent vectors. A theoretical explanation for this phenomenon establishes that high capacity autoencoders can learn an arbitrary mapping between sequences and associated latent representations. To remedy this issue, we augment adversarial autoencoders with a denoising objective where original sentences are reconstructed from perturbed versions (referred to as DAAE). We prove that this simple modification guides the latent space geometry of the resulting model by encouraging the encoder to map similar texts to similar latent representations. In empirical comparisons with various types of autoencoders, our model provides the best trade-off between generation quality and reconstruction capacity. Moreover, the improved geometry of the DAAE latent space enables zero-shot text style transfer via simple latent vector arithmetic.

arXiv commentICML 2020 camera-ready

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Style Transfer from Non-Parallel Text by Cross-Alignment
• Universal Transformers
• InfoVAE: Information Maximizing Variational Autoencoders
• Phrase-Based & Neural Unsupervised Machine Translation
• Generalized Denoising Auto-Encoders as Generative Models
• Unsupervised Machine Translation Using Monolingual Corpora Only
• Wasserstein Auto-Encoders
• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
• Unsupervised Neural Machine Translation
• Content preserving text generation with attribute controls
• Denoising Criterion for Variational Auto-Encoding Framework
• Analyzing noise in autoencoders and deep networks
• Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference
• Multiple-Attribute Text Style Transfer
• Semi-Amortized Variational Autoencoders
• Eval all, trust a few, do wrong to none: Comparing sentence generation models
• On the Latent Space of Wasserstein Auto-Encoders