Sequence Level Training with Recurrent Neural Networks

Abstract

Many natural language processing applications use language models to generate text. These models are typically trained to predict the next word in a sequence, given the previous words and some context such as an image. However, at test time the model is expected to generate the entire sequence from scratch. This discrepancy makes generation brittle, as errors may accumulate along the way. We address this issue by proposing a novel sequence level training algorithm that directly optimizes the metric used at test time, such as BLEU or ROUGE. On three different tasks, our approach outperforms several strong baselines for greedy generation. The method is also competitive when these baselines employ beam search, while being several times faster.

References in corpus (2)

• Sequence to Sequence Learning with Neural Networks
• Recurrent Models of Visual Attention

Cited by in corpus (195)

• A Brief Survey of Deep Reinforcement Learning
• Neural Architecture Search with Reinforcement Learning
• An Introduction to Deep Reinforcement Learning
• A Deep Reinforced Model for Abstractive Summarization
• Recent Trends in Deep Learning Based Natural Language Processing
• Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Models
• Understanding Neural Networks through Representation Erasure
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Deep Reinforcement Learning for Dialogue Generation
• Fine-Tuning Language Models from Human Preferences
• SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
• Neural Text Generation with Unlikelihood Training
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• An Actor-Critic Algorithm for Sequence Prediction
• Natural Language Processing Advancements By Deep Learning: A Survey
• Context-Aware Visual Policy Network for Fine-Grained Image Captioning
• Modeling Human Motion with Quaternion-based Neural Networks
• Context-Aware Visual Policy Network for Sequence-Level Image Captioning
• Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory
• A Teacher-Student Framework for Zero-Resource Neural Machine Translation
• Adversarial Generation of Natural Language
• MeanSum: A Neural Model for Unsupervised Multi-document Abstractive Summarization
• CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning
• A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
• Language Generation with Recurrent Generative Adversarial Networks without Pre-training
• Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model
• Neural Models for Information Retrieval
• Dialog-based Language Learning
• RNGDet: Road Network Graph Detection by Transformer in Aerial Images
• Language GANs Falling Short
• Toward Subgraph-Guided Knowledge Graph Question Generation with Graph Neural Networks
• Adversarial Neural Machine Translation
• Neural Abstractive Text Summarization with Sequence-to-Sequence Models
• Convolutional Hierarchical Attention Network for Query-Focused Video Summarization
• Deep Reinforcement Learning-based Image Captioning with Embedding Reward
• A Convolutional Encoder Model for Neural Machine Translation
• XNMT: The eXtensible Neural Machine Translation Toolkit
• Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning
• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• Video Storytelling: Textual Summaries for Events
• Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model
• Reinforcement Learning-powered Semantic Communication via Semantic Similarity
• Predicting the Popularity of Micro-videos with Multimodal Variational Encoder-Decoder Framework
• Minimum Risk Training for Neural Machine Translation
• Controlling Output Length in Neural Encoder-Decoders
• Accurate Machine Learning Atmospheric Retrieval via a Neural Network Surrogate Model for Radiative Transfer
• Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
• Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
• Towards Binary-Valued Gates for Robust LSTM Training
• From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood
• Neural Headline Generation with Sentence-wise Optimization
• Learning to summarize from human feedback
• Unifying Multimodal Transformer for Bi-directional Image and Text Generation
• Multi-Hop Knowledge Graph Reasoning with Reward Shaping
• Table2Charts: Recommending Charts by Learning Shared Table Representations
• Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models
• Teaching Machines to Describe Images via Natural Language Feedback
• Motion Generation Using Bilateral Control-Based Imitation Learning with Autoregressive Learning
• Description Based Text Classification with Reinforcement Learning
• Learning to Extract Coherent Summary via Deep Reinforcement Learning
• Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training
• Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner
• Auto-Encoding Scene Graphs for Image Captioning
• A Reinforced Topic-Aware Convolutional Sequence-to-Sequence Model for Abstractive Text Summarization
• Transcribing Content from Structural Images with Spotlight Mechanism
• Less Is More: Picking Informative Frames for Video Captioning
• Explanation as a Defense of Recommendation
• A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss
• Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control
• Neural Language Generation: Formulation, Methods, and Evaluation
• No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling
• Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation
• Rethinking the Reference-based Distinctive Image Captioning
• Video Captioning via Hierarchical Reinforcement Learning
• Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generation
• CoT: Cooperative Training for Generative Modeling of Discrete Data
• Non-Autoregressive Machine Translation with Auxiliary Regularization
• Data Distillation for Controlling Specificity in Dialogue Generation
• Hierarchical LSTMs with Adaptive Attention for Visual Captioning
• Deep Reinforcement Learning for Market Making Under a Hawkes Process-Based Limit Order Book Model
• Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input
• Heterogeneous Interaction Modeling With Reduced Accumulated Error for Multi-Agent Trajectory Prediction
• Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation
• Trainable Greedy Decoding for Neural Machine Translation
• Ad Headline Generation using Self-Critical Masked Language Model
• Deep Reinforced Query Reformulation for Information Retrieval
• An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generation
• Natural Question Generation with Reinforcement Learning Based Graph-to-Sequence Model
• Improving Neural Machine Translation with Conditional Sequence Generative Adversarial Nets
• Multimodal Machine Translation with Reinforcement Learning
• Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation
• Action Assembly: Sparse Imitation Learning for Text Based Games with Combinatorial Action Spaces
• Putting the Horse Before the Cart:A Generator-Evaluator Framework for Question Generation from Text
• CREDIT: Coarse-to-Fine Sequence Generation for Dialogue State Tracking
• The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction
• Debiasing the Cloze Task in Sequential Recommendation with Bidirectional Transformers
• Joint Training of Candidate Extraction and Answer Selection for Reading Comprehension
• Differentiable Weighted Finite-State Transducers
• Grammatical Error Correction with Neural Reinforcement Learning
• Answers Unite! Unsupervised Metrics for Reinforced Summarization Models
• On the Weaknesses of Reinforcement Learning for Neural Machine Translation
• Defending Against Backdoor Attacks in Natural Language Generation
• Reinforcement Learning for Weakly Supervised Temporal Grounding of Natural Language in Untrimmed Videos
• Does Simultaneous Speech Translation need Simultaneous Models?
• Discriminative Adversarial Search for Abstractive Summarization
• Translating Math Formula Images to LaTeX Sequences Using Deep Neural Networks with Sequence-level Training
• Evaluating Transfer Learning for Simplifying GitHub READMEs
• ACtuAL: Actor-Critic Under Adversarial Learning
• Hard but Robust, Easy but Sensitive: How Encoder and Decoder Perform in Neural Machine Translation
• Regularizing Neural Machine Translation by Target-bidirectional Agreement
• Fine-Grained Image Captioning with Global-Local Discriminative Objective
• Large-scale Pretraining for Neural Machine Translation with Tens of Billions of Sentence Pairs
• ReWE: Regressing Word Embeddings for Regularization of Neural Machine Translation Systems
• Beyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter
• Discourse-Aware Neural Rewards for Coherent Text Generation
• Improving End-to-End Speech Recognition with Policy Learning
• Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task
• Machine Translation : From Statistical to modern Deep-learning practices
• Greedy Search with Probabilistic N-gram Matching for Neural Machine Translation
• ColdGANs: Taming Language GANs with Cautious Sampling Strategies
• A New GAN-based End-to-End TTS Training Algorithm
• Tag-less Back-Translation
• Zero-Resource Neural Machine Translation with Multi-Agent Communication Game
• Dialog State Tracking with Reinforced Data Augmentation
• Finding It at Another Side: A Viewpoint-Adapted Matching Encoder for Change Captioning
• On the Inference Calibration of Neural Machine Translation
• A Deep Reinforced Sequence-to-Set Model for Multi-Label Text Classification
• APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
• Proximal Policy Optimization and its Dynamic Version for Sequence Generation
• Evaluating Rewards for Question Generation Models
• A Pilot Study of Domain Adaptation Effect for Neural Abstractive Summarization
• Teaching Machines to Converse
• Topo-boundary: A Benchmark Dataset on Topological Road-boundary Detection Using Aerial Images for Autonomous Driving
• Multimedia Generative Script Learning for Task Planning
• Non-parallel Voice Conversion System with WaveNet Vocoder and Collapsed Speech Suppression
• A Stable and Effective Learning Strategy for Trainable Greedy Decoding
• In-Home Daily-Life Captioning Using Radio Signals
• SHAPED: Shared-Private Encoder-Decoder for Text Style Adaptation
• Informative Image Captioning with External Sources of Information
• Dense Information Flow for Neural Machine Translation
• How Local is the Local Diversity? Reinforcing Sequential Determinantal Point Processes with Dynamic Ground Sets for Supervised Video Summarization
• Distilling Knowledge for Search-based Structured Prediction
• Reinforcement Learning of Graph Neural Networks for Service Function Chaining
• Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning
• LeafNATS: An Open-Source Toolkit and Live Demo System for Neural Abstractive Text Summarization
• Energy-Based Models with Applications to Speech and Language Processing
• Goal-directed Generation of Discrete Structures with Conditional Generative Models
• ECOL-R: Encouraging Copying in Novel Object Captioning with Reinforcement Learning
• AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks
• Towards Amortized Ranking-Critical Training for Collaborative Filtering
• Boosting Naturalness of Language in Task-oriented Dialogues via Adversarial Training
• Backprop-Q: Generalized Backpropagation for Stochastic Computation Graphs
• Improving Sequential Determinantal Point Processes for Supervised Video Summarization
• An Overview of Natural Language State Representation for Reinforcement Learning
• From Credit Assignment to Entropy Regularization: Two New Algorithms for Neural Sequence Prediction
• Text Generation Based on Generative Adversarial Nets with Latent Variable
• BanditRank: Learning to Rank Using Contextual Bandits
• Towards one-shot learning for rare-word translation with external experts
• Improving Reinforcement Learning Based Image Captioning with Natural Language Prior
• Intention Oriented Image Captions with Guiding Objects
• Promising Accurate Prefix Boosting for sequence-to-sequence ASR
• Learning Domain Adaptation with Model Calibration for Surgical Report Generation in Robotic Surgery
• Paraphrase Generation as Unsupervised Machine Translation
• Sequence-to-Sequence ASR Optimization via Reinforcement Learning
• Controllable Length Control Neural Encoder-Decoder via Reinforcement Learning
• Neural Machine Translation: A Review and Survey
• Mining for meaning: from vision to language through multiple networks consensus
• Paraphrases as Foreign Languages in Multilingual Neural Machine Translation
• Transferring Source Style in Non-Parallel Voice Conversion
• Seq-SG2SL: Inferring Semantic Layout from Scene Graph Through Sequence to Sequence Learning
• Bayesian Attention Modules
• Transfer Reward Learning for Policy Gradient-Based Text Generation
• $k$-Neighbor Based Curriculum Sampling for Sequence Prediction
• A Differentially Private Multi-Output Deep Generative Networks Approach For Activity Diary Synthesis
• Context-Dependent Semantic Parsing over Temporally Structured Data
• Differentiable Sampling with Flexible Reference Word Order for Neural Machine Translation
• Enhance Long Text Understanding via Distilled Gist Detector from Abstractive Summarization
• SparseGAN: Sparse Generative Adversarial Network for Text Generation
• Towards Reinforcement Learning for Pivot-based Neural Machine Translation with Non-autoregressive Transformer
• When Better Eyes Lead to Blindness: A Diagnostic Study of the Information Bottleneck in CNN-LSTM Image Captioning Models
• A Face-to-Face Neural Conversation Model
• TAG : Type Auxiliary Guiding for Code Comment Generation
• Boosting Summarization with Normalizing Flows and Aggressive Training
• Predict and Use Latent Patterns for Short-Text Conversation
• Improving Automatic Source Code Summarization via Deep Reinforcement Learning
• Boosting Image Recognition with Non-differentiable Constraints
• The Highs and Lows of Simple Lexical Domain Adaptation Approaches for Neural Machine Translation
• Stack-VS: Stacked Visual-Semantic Attention for Image Caption Generation
• Multitasking Inhibits Semantic Drift
• Quantity vs. Quality of Monolingual Source Data in Automatic Text Translation: Can It Be Too Little If It Is Too Good?
• Integrating User and Agent Models: A Deep Task-Oriented Dialogue System
• Approximate Distribution Matching for Sequence-to-Sequence Learning
• Ranking sentences from product description & bullets for better search
• Learn to Talk via Proactive Knowledge Transfer
• Autoregressive Knowledge Distillation through Imitation Learning