Do sequence-to-sequence VAEs learn global features of sentences?

Abstract

Autoregressive language models are powerful and relatively easy to train. However, these models are usually trained without explicit conditioning labels and do not offer easy ways to control global aspects such as sentiment or topic during generation. Bowman & al. (2016) adapted the Variational Autoencoder (VAE) for natural language with the sequence-to-sequence architecture and claimed that the latent vector was able to capture such global features in an unsupervised manner. We question this claim. We measure which words benefit most from the latent information by decomposing the reconstruction loss per position in the sentence. Using this method, we find that VAEs are prone to memorizing the first words and the sentence length, producing local features of limited usefulness. To alleviate this, we investigate alternative architectures based on bag-of-words assumptions and language model pretraining. These variants learn latent variables that are more global, i.e., more predictive of topic or sentiment labels. Moreover, using reconstructions, we observe that they decrease memorization: the first word and the sentence length are not recovered as accurately than with the baselines, consequently yielding more diverse reconstructions.

arXiv commentCamera-ready version, EMNLP2020

References in corpus (18)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Auto-Encoding Variational Bayes
• Sequence to Sequence Learning with Neural Networks
• Character-level Convolutional Networks for Text Classification
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Thumbs up? Sentiment Classification using Machine Learning Techniques
• Semi-Supervised Learning with Deep Generative Models
• Non-Autoregressive Neural Machine Translation
• Variational Lossy Autoencoder
• Importance Weighted Autoencoders
• Neural Variational Inference for Text Processing
• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
• Denoising Criterion for Variational Auto-Encoding Framework
• RNNs Implicitly Implement Tensor Product Representations
• Latent Normalizing Flows for Discrete Sequences
• Conditional Variational Autoencoder for Neural Machine Translation
• Hierarchical Autoregressive Image Models with Auxiliary Decoders
• Unsupervised Abstractive Sentence Summarization using Length Controlled Variational Autoencoder

Cited by in corpus (2)

• SentenceMIM: A Latent Variable Language Model
• The Neglected Sibling: Isotropic Gaussian Posterior for VAE