Neural Text Generation: A Practical Guide

Abstract

Deep learning methods have recently achieved great empirical success on machine translation, dialogue response generation, summarization, and other text generation tasks. At a high level, the technique has been to train end-to-end neural network models consisting of an encoder model to produce a hidden representation of the source text, followed by a decoder model to generate the target. While such models have significantly fewer pieces than earlier systems, significant tuning is still required to achieve good performance. For text generation models in particular, the decoder can behave in undesired ways, such as by generating truncated or repetitive outputs, outputting bland and generic responses, or in some cases producing ungrammatical gibberish. This paper is intended as a practical guide for resolving such undesired behavior in text generation models, with the aim of helping enable real-world applications.

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Neural Machine Translation by Jointly Learning to Align and Translate
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Recurrent Neural Network Regularization
• Exploring the Limits of Language Modeling
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Deep Reinforcement Learning for Dialogue Generation
• Neural Machine Translation in Linear Time
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• An Actor-Critic Algorithm for Sequence Prediction
• Adversarial Learning for Neural Dialogue Generation
• Data Noising as Smoothing in Neural Network Language Models
• Modeling Coverage for Neural Machine Translation
• Get To The Point: Summarization with Pointer-Generator Networks
• Addressing the Rare Word Problem in Neural Machine Translation
• Massive Exploration of Neural Machine Translation Architectures
• On the State of the Art of Evaluation in Neural Language Models

Cited by in corpus (7)

• Clinically Accurate Chest X-Ray Report Generation
• Neural Text Generation: Past, Present and Beyond
• PaperRobot: Incremental Draft Generation of Scientific Ideas
• How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG
• Beta DVBF: Learning State-Space Models for Control from High Dimensional Observations
• Keeping Notes: Conditional Natural Language Generation with a Scratchpad Mechanism
• AffectON: Incorporating Affect Into Dialog Generation