Language Generation with Recurrent Generative Adversarial Networks without Pre-training

Abstract

Generative Adversarial Networks (GANs) have shown great promise recently in image generation. Training GANs for language generation has proven to be more difficult, because of the non-differentiable nature of generating text with recurrent neural networks. Consequently, past work has either resorted to pre-training with maximum-likelihood or used convolutional networks for generation. In this work, we show that recurrent neural networks can be trained to generate text with GANs from scratch using curriculum learning, by slowly teaching the model to generate sequences of increasing and variable length. We empirically show that our approach vastly improves the quality of generated sequences compared to a convolutional baseline.

Presented at the 1st Workshop on Learning to Generate Natural Language at ICML 2017

References in corpus (12)

• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Improved Training of Wasserstein GANs
• Sequence Level Training with Recurrent Neural Networks
• Exploring the Limits of Language Modeling
• SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
• Adversarial Learning for Neural Dialogue Generation
• Maximum-Likelihood Augmented Discrete Generative Adversarial Networks
• Boundary-Seeking Generative Adversarial Networks
• Recurrent Topic-Transition GAN for Visual Paragraph Generation
• Adversarial Neural Machine Translation
• Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training

Cited by in corpus (37)

• MaskGAN: Better Text Generation via Filling in the______
• Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities
• Fast and Accurate Optical Fiber Channel Modeling Using Generative Adversarial Network
• RNNIDS: Enhancing Network Intrusion Detection Systems through Deep Learning
• Probabilistic Forecasting of Sensory Data with Generative Adversarial Networks - ForGAN
• Human Motion Modeling using DVGANs
• Adversarially Regularized Autoencoders
• A Tutorial on Deep Latent Variable Models of Natural Language
• Airline Passenger Name Record Generation using Generative Adversarial Networks
• Low-Resource Text Classification using Domain-Adversarial Learning
• Neural Language Generation: Formulation, Methods, and Evaluation
• SPI-GAN: Towards Single-Pixel Imaging through Generative Adversarial Network
• On the Weaknesses of Reinforcement Learning for Neural Machine Translation
• Improved Training with Curriculum GANs
• On the Effectiveness of Pretrained Models for API Learning
• Generating Realistic Stock Market Order Streams
• Deep Learning Based on Generative Adversarial and Convolutional Neural Networks for Financial Time Series Predictions
• LD-SDS: Towards an Expressive Spoken Dialogue System based on Linked-Data
• Improving Conditional Sequence Generative Adversarial Networks by Stepwise Evaluation
• Artificial Intelligence in Intelligent Tutoring Robots: A Systematic Review and Design Guidelines
• Snore-GANs: Improving Automatic Snore Sound Classification with Synthesized Data
• Extractive Summary as Discrete Latent Variables
• Sparse Text Generation
• Unsupervised Text Embedding Space Generation Using Generative Adversarial Networks for Text Synthesis
• Pipeline Generative Adversarial Networks for Facial Images Generation with Multiple Attributes
• Adversarial Learning of Semantic Relevance in Text to Image Synthesis
• Collaborative Training of GANs in Continuous and Discrete Spaces for Text Generation
• Curriculum Learning for Deep Generative Models with Clustering
• Classification of sparsely labeled spatio-temporal data through semi-supervised adversarial learning
• Language Modeling with Generative Adversarial Networks
• A Differentially Private Multi-Output Deep Generative Networks Approach For Activity Diary Synthesis
• Instance Map based Image Synthesis with a Denoising Generative Adversarial Network
• Improving Model Compatibility of Generative Adversarial Networks by Boundary Calibration
• Modeling Coherency in Generated Emails by Leveraging Deep Neural Learners
• Can adversarial training learn image captioning ?
• ReshapeGAN: Object Reshaping by Providing A Single Reference Image
• Generative Adversarial Nets for Multiple Text Corpora