MaskGAN: Better Text Generation via Filling in the______

Abstract

Neural text generation models are often autoregressive language models or seq2seq models. These models generate text by sampling words sequentially, with each word conditioned on the previous word, and are state-of-the-art for several machine translation and summarization benchmarks. These benchmarks are often defined by validation perplexity even though this is not a direct measure of the quality of the generated text. Additionally, these models are typically trained via maxi- mum likelihood and teacher forcing. These methods are well-suited to optimizing perplexity but can result in poor sample quality since generating text requires conditioning on sequences of words that may have never been observed at training time. We propose to improve sample quality using Generative Adversarial Networks (GANs), which explicitly train the generator to produce high quality samples and have shown a lot of success in image generation. GANs were originally designed to output differentiable values, so discrete language generation is challenging for them. We claim that validation perplexity alone is not indicative of the quality of text generated by a model. We introduce an actor-critic conditional GAN that fills in missing text conditioned on the surrounding context. We show qualitatively and quantitatively, evidence that this produces more realistic conditional and unconditional text samples compared to a maximum likelihood trained model.

arXiv comment16 pages, ICLR 2018

References in corpus (4)

• Maximum-Likelihood Augmented Discrete Generative Adversarial Networks
• Adversarial Feature Matching for Text Generation
• Boundary-Seeking Generative Adversarial Networks
• Language Generation with Recurrent Generative Adversarial Networks without Pre-training

Cited by in corpus (97)

• Generative Adversarial Network in Medical Imaging: A Review
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Deep learning for determining a near-optimal topological design without any iteration
• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• Synthesizing Tabular Data using Generative Adversarial Networks
• Using GANs for Sharing Networked Time Series Data: Challenges, Initial Promise, and Open Questions
• Clinically Accurate Chest X-Ray Report Generation
• A survey on text generation using generative adversarial networks
• Language GANs Falling Short
• On Accurate Evaluation of GANs for Language Generation
• Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• On Leveraging Pretrained GANs for Generation with Limited Data
• Neural Text Generation: Past, Present and Beyond
• Actor-Critic based Training Framework for Abstractive Summarization
• Creativity and Machine Learning: A Survey
• Small-GAN: Speeding Up GAN Training Using Core-sets
• Regularization Methods for Generative Adversarial Networks: An Overview of Recent Studies
• Generative Adversarial Networks: A Survey Towards Private and Secure Applications
• Improving Sequence-to-Sequence Learning via Optimal Transport
• Neural Language Generation: Formulation, Methods, and Evaluation
• House-GAN++: Generative Adversarial Layout Refinement Networks
• Some Theoretical Insights into Wasserstein GANs
• Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation
• Bayesian Inference with Generative Adversarial Network Priors
• Adversarial System Variant Approximation to Quantify Process Model Generalization
• Connecting the Dots Between MLE and RL for Sequence Prediction
• Correlated discrete data generation using adversarial training
• New Era of Deeplearning-Based Malware Intrusion Detection: The Malware Detection and Prediction Based On Deep Learning
• Creative GANs for generating poems, lyrics, and metaphors
• Towards Generating Long and Coherent Text with Multi-Level Latent Variable Models
• Improved Training with Curriculum GANs
• Improving GAN Training with Probability Ratio Clipping and Sample Reweighting
• Unsupervised Image Captioning
• Enabling Language Models to Fill in the Blanks
• Topic-Preserving Synthetic News Generation: An Adversarial Deep Reinforcement Learning Approach
• On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models
• Multi-Modal Generative Adversarial Network for Short Product Title Generation in Mobile E-Commerce
• BERT as a Teacher: Contextual Embeddings for Sequence-Level Reward
• Open Event Extraction from Online Text using a Generative Adversarial Network
• ColdGANs: Taming Language GANs with Cautious Sampling Strategies
• Neural Topic Modeling with Bidirectional Adversarial Training
• Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation
• ARAML: A Stable Adversarial Training Framework for Text Generation
• The Automated Inspection of Opaque Liquid Vaccines
• Efficient, Simple and Automated Negative Sampling for Knowledge Graph Embedding
• Extractive Summary as Discrete Latent Variables
• Conditional Hybrid GAN for Sequence Generation
• BasketballGAN: Generating Basketball Play Simulation Through Sketching
• GAN Memory with No Forgetting
• Adding A Filter Based on The Discriminator to Improve Unconditional Text Generation
• Automated Source Code Generation and Auto-completion Using Deep Learning: Comparing and Discussing Current Language-Model-Related Approaches
• Narrative Incoherence Detection
• GANs with Variational Entropy Regularizers: Applications in Mitigating the Mode-Collapse Issue
• Semi-supervised Conditional GAN for Simultaneous Generation and Detection of Phishing URLs: A Game theoretic Perspective
• Time-series Imputation of Temporally-occluded Multiagent Trajectories
• Controlling Text Edition by Changing Answers of Specific Questions
• Differential-Critic GAN: Generating What You Want by a Cue of Preferences
• Text Generation with Deep Variational GAN
• GMM-Based Generative Adversarial Encoder Learning
• A Novel Framework for Selection of GANs for an Application
• Quantum Optical Experiments Modeled by Long Short-Term Memory
• A Neuro-AI Interface for Evaluating Generative Adversarial Networks
• Towards Coherent and Cohesive Long-form Text Generation
• Adversarial Sub-sequence for Text Generation
• Latent Adversarial Defence with Boundary-guided Generation
• Leveraging GPT-2 for Classifying Spam Reviews with Limited Labeled Data via Adversarial Training
• SparseGAN: Sparse Generative Adversarial Network for Text Generation
• Towards a Universal NLG for Dialogue Systems and Simulators with Future Bridging
• Russian Natural Language Generation: Creation of a Language Modelling Dataset and Evaluation with Modern Neural Architectures
• Protecting Anonymous Speech: A Generative Adversarial Network Methodology for Removing Stylistic Indicators in Text
• Distributional Discrepancy: A Metric for Unconditional Text Generation
• ATM:Adversarial-neural Topic Model
• Spelling Correction with Denoising Transformer
• Generative Models for Security: Attacks, Defenses, and Opportunities
• Evaluating a Generative Adversarial Framework for Information Retrieval
• EMIXER: End-to-end Multimodal X-ray Generation via Self-supervision
• Plan ahead: Self-Supervised Text Planning for Paragraph Completion Task
• ReGAN: RE[LAX|BAR|INFORCE] based Sequence Generation using GANs
• Language Modeling with Generative Adversarial Networks
• Learning Rhyming Constraints using Structured Adversaries
• Measure-conditional Discriminator with Stationary Optimum for GANs and Statistical Distance Surrogates
• MaskCycleGAN-VC: Learning Non-parallel Voice Conversion with Filling in Frames
• Adversarial Machine Learning in Text Analysis and Generation
• Inspiration through Observation: Demonstrating the Influence of Automatically Generated Text on Creative Writing
• Diamonds in the Rough: Generating Fluent Sentences from Early-Stage Drafts for Academic Writing Assistance
• Emotion Detection with Neural Personal Discrimination
• TaylorGAN: Neighbor-Augmented Policy Update for Sample-Efficient Natural Language Generation
• The Impact of Hole Geometry on Relative Robustness of In-Painting Networks: An Empirical Study
• Generative Adversarial Networks for Annotated Data Augmentation in Data Sparse NLU
• Depthwise Discrete Representation Learning
• AriEL: volume coding for sentence generation
• Generating Symbolic Reasoning Problems with Transformer GANs
• Diversifying Topic-Coherent Response Generation for Natural Multi-turn Conversations
• Token Manipulation Generative Adversarial Network for Text Generation
• Efficient text generation of user-defined topic using generative adversarial networks