mu-Forcing: Training Variational Recurrent Autoencoders for Text Generation
arXiv:1905.10072 · doi:10.1145/3341110
Abstract
It has been previously observed that training Variational Recurrent Autoencoders (VRAE) for text generation suffers from serious uninformative latent variables problem. The model would collapse into a plain language model that totally ignore the latent variables and can only generate repeating and dull samples. In this paper, we explore the reason behind this issue and propose an effective regularizer based approach to address it. The proposed method directly injects extra constraints on the posteriors of latent variables into the learning process of VRAE, which can flexibly and stably control the trade-off between the KL term and the reconstruction term, making the model learn dense and meaningful latent representations. The experimental results show that the proposed method outperforms several strong baselines and can make the model learn interpretable latent variables and generate diverse meaningful sentences. Furthermore, the proposed method can perform well without using other strategies, such as KL annealing.
arXiv commentTo appear in the ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP)
References in corpus (19)
- Adam: A Method for Stochastic Optimization
- Auto-Encoding Variational Bayes
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Variational Inference with Normalizing Flows
- DRAW: A Recurrent Neural Network For Image Generation
- Layer Normalization
- Semi-supervised Sequence Learning
- InfoVAE: Information Maximizing Variational Autoencoders
- Variational Lossy Autoencoder
- Ladder Variational Autoencoders
- Improving Variational Inference with Inverse Autoregressive Flow
- Variational Recurrent Auto-Encoders
- Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders
- Texygen: A Benchmarking Platform for Text Generation Models
- Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
- Tackling Over-pruning in Variational Autoencoders
- Semi-Amortized Variational Autoencoders
- Z-Forcing: Training Stochastic Recurrent Networks
- Improving Variational Encoder-Decoders in Dialogue Generation