Learning Natural Language Generation from Scratch

Abstract

This paper introduces TRUncated ReinForcement Learning for Language (TrufLL), an original ap-proach to train conditional language models from scratch by only using reinforcement learning (RL). AsRL methods unsuccessfully scale to large action spaces, we dynamically truncate the vocabulary spaceusing a generic language model. TrufLL thus enables to train a language agent by solely interacting withits environment without any task-specific prior knowledge; it is only guided with a task-agnostic languagemodel. Interestingly, this approach avoids the dependency to labelled datasets and inherently reduces pre-trained policy flaws such as language or exposure biases. We evaluate TrufLL on two visual questiongeneration tasks, for which we report positive results over performance and language metrics, which wethen corroborate with a human evaluation. To our knowledge, it is the first approach that successfullylearns a language generation policy (almost) from scratch.

References in corpus (17)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Categorical Reparameterization with Gumbel-Softmax
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Deep Reinforcement Learning for Dialogue Generation
• Fine-Tuning Language Models from Human Preferences
• Optimizing Dialogue Management with Reinforcement Learning: Experiments with the NJFun System
• A Diversity-Promoting Objective Function for Neural Conversation Models
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning
• CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog
• Heated-Up Softmax Embedding
• Learning to summarize from human feedback
• Trading Off Diversity and Quality in Natural Language Generation
• Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learning
• Unsupervised Contextual Paraphrase Generation using Lexical Control and Reinforcement Learning
• Keep CALM and Explore: Language Models for Action Generation in Text-based Games
• Contextual Temperature for Language Modeling