TaylorGAN: Neighbor-Augmented Policy Update for Sample-Efficient Natural Language Generation

Abstract

Score function-based natural language generation (NLG) approaches such as REINFORCE, in general, suffer from low sample efficiency and training instability problems. This is mainly due to the non-differentiable nature of the discrete space sampling and thus these methods have to treat the discriminator as a black box and ignore the gradient information. To improve the sample efficiency and reduce the variance of REINFORCE, we propose a novel approach, TaylorGAN, which augments the gradient estimation by off-policy update and the first-order Taylor expansion. This approach enables us to train NLG models from scratch with smaller batch size -- without maximum likelihood pre-training, and outperforms existing GAN-based methods on multiple metrics of quality and diversity. The source code and data are available at https://github.com/MiuLab/TaylorGAN

arXiv comment34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Categorical Reparameterization with Gumbel-Softmax
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• MaskGAN: Better Text Generation via Filling in the______
• Release Strategies and the Social Impacts of Language Models
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution
• Maximum-Likelihood Augmented Discrete Generative Adversarial Networks
• Adversarial Ranking for Language Generation
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• Language GANs Falling Short
• Prescribed Generative Adversarial Networks
• Lipschitz Generative Adversarial Nets
• Policy Optimization Reinforcement Learning with Entropy Regularization