Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space
arXiv:2004.04092
Abstract
When trained effectively, the Variational Autoencoder (VAE) can be both a powerful generative model and an effective representation learning framework for natural language. In this paper, we propose the first large-scale language VAE model, Optimus. A universal latent embedding space for sentences is first pre-trained on large text corpus, and then fine-tuned for various language generation and understanding tasks. Compared with GPT-2, Optimus enables guided language generation from an abstract level using the latent vectors. Compared with BERT, Optimus can generalize better on low-resource language understanding tasks due to the smooth latent space structure. Extensive experimental results on a wide range of language tasks demonstrate the effectiveness of Optimus. It achieves new state-of-the-art on VAE language modeling benchmarks. We hope that our first pre-trained big VAE language model itself and results can help the NLP community renew the interests of deep generative models in the era of large-scale pre-training, and make these principled methods more practical.
arXiv commentAccepted in EMNLP 2020; Code: https://github.com/ChunyuanLI/Optimus Demo: http://aka.ms/optimus
References in corpus (11)
- Conditional Generative Adversarial Nets
- Distributed Representations of Sentences and Documents
- The information bottleneck method
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- Skip-Thought Vectors
- Style Transfer from Non-Parallel Text by Cross-Alignment
- Semi-supervised Sequence Learning
- Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing
- Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
- A Tutorial on Deep Latent Variable Models of Natural Language