Sequence Level Training with Recurrent Neural Networks

Abstract

Many natural language processing applications use language models to generate text. These models are typically trained to predict the next word in a sequence, given the previous words and some context such as an image. However, at test time the model is expected to generate the entire sequence from scratch. This discrepancy makes generation brittle, as errors may accumulate along the way. We address this issue by proposing a novel sequence level training algorithm that directly optimizes the metric used at test time, such as BLEU or ROUGE. On three different tasks, our approach outperforms several strong baselines for greedy generation. The method is also competitive when these baselines employ beam search, while being several times faster.

References in corpus (5)

• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Building Natural-Language Generation Systems
• Recurrent Models of Visual Attention

Cited by in corpus (196)

• A Brief Survey of Deep Reinforcement Learning
• Neural Architecture Search with Reinforcement Learning
• An Introduction to Deep Reinforcement Learning
• A Deep Reinforced Model for Abstractive Summarization
• Recent Trends in Deep Learning Based Natural Language Processing
• Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Models
• Understanding Neural Networks through Representation Erasure
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Deep Reinforcement Learning for Dialogue Generation
• Fine-Tuning Language Models from Human Preferences
• SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
• Neural Text Generation with Unlikelihood Training
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• An Actor-Critic Algorithm for Sequence Prediction
• Natural Language Processing Advancements By Deep Learning: A Survey
• Context-Aware Visual Policy Network for Fine-Grained Image Captioning
• Modeling Human Motion with Quaternion-based Neural Networks
• Context-Aware Visual Policy Network for Sequence-Level Image Captioning
• Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory
• A Teacher-Student Framework for Zero-Resource Neural Machine Translation
• Adversarial Generation of Natural Language
• MeanSum: A Neural Model for Unsupervised Multi-document Abstractive Summarization
• CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning
• A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
• Language Generation with Recurrent Generative Adversarial Networks without Pre-training
• Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
• Neural Models for Information Retrieval
• Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model
• Dialog-based Language Learning
• RNGDet: Road Network Graph Detection by Transformer in Aerial Images
• Language GANs Falling Short
• Toward Subgraph-Guided Knowledge Graph Question Generation with Graph Neural Networks
• Adversarial Neural Machine Translation
• Convolutional Hierarchical Attention Network for Query-Focused Video Summarization
• Neural Abstractive Text Summarization with Sequence-to-Sequence Models
• Deep Reinforcement Learning-based Image Captioning with Embedding Reward
• A Convolutional Encoder Model for Neural Machine Translation
• XNMT: The eXtensible Neural Machine Translation Toolkit
• Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning
• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• Video Storytelling: Textual Summaries for Events
• Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model
• Predicting the Popularity of Micro-videos with Multimodal Variational Encoder-Decoder Framework
• Reinforcement Learning-powered Semantic Communication via Semantic Similarity
• Minimum Risk Training for Neural Machine Translation
• Accurate Machine Learning Atmospheric Retrieval via a Neural Network Surrogate Model for Radiative Transfer
• Controlling Output Length in Neural Encoder-Decoders
• Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
• Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
• Towards Binary-Valued Gates for Robust LSTM Training
• From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood
• Neural Headline Generation with Sentence-wise Optimization
• Learning to summarize from human feedback
• Unifying Multimodal Transformer for Bi-directional Image and Text Generation
• Multi-Hop Knowledge Graph Reasoning with Reward Shaping
• Table2Charts: Recommending Charts by Learning Shared Table Representations
• Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models
• Teaching Machines to Describe Images via Natural Language Feedback
• Motion Generation Using Bilateral Control-Based Imitation Learning with Autoregressive Learning
• Description Based Text Classification with Reinforcement Learning
• Learning to Extract Coherent Summary via Deep Reinforcement Learning
• Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner
• Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training
• A Reinforced Topic-Aware Convolutional Sequence-to-Sequence Model for Abstractive Text Summarization
• Less Is More: Picking Informative Frames for Video Captioning
• Auto-Encoding Scene Graphs for Image Captioning
• Transcribing Content from Structural Images with Spotlight Mechanism
• Explanation as a Defense of Recommendation
• A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss
• Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control
• Neural Language Generation: Formulation, Methods, and Evaluation
• No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling
• Rethinking the Reference-based Distinctive Image Captioning
• Video Captioning via Hierarchical Reinforcement Learning
• Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation
• CoT: Cooperative Training for Generative Modeling of Discrete Data
• Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generation
• Data Distillation for Controlling Specificity in Dialogue Generation
• Non-Autoregressive Machine Translation with Auxiliary Regularization
• Hierarchical LSTMs with Adaptive Attention for Visual Captioning
• Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input
• Deep Reinforcement Learning for Market Making Under a Hawkes Process-Based Limit Order Book Model
• Heterogeneous Interaction Modeling With Reduced Accumulated Error for Multi-Agent Trajectory Prediction
• Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation
• Trainable Greedy Decoding for Neural Machine Translation
• Ad Headline Generation using Self-Critical Masked Language Model
• An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generation
• Natural Question Generation with Reinforcement Learning Based Graph-to-Sequence Model
• Deep Reinforced Query Reformulation for Information Retrieval
• Improving Neural Machine Translation with Conditional Sequence Generative Adversarial Nets
• Multimodal Machine Translation with Reinforcement Learning
• Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation
• Putting the Horse Before the Cart:A Generator-Evaluator Framework for Question Generation from Text
• The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction
• Action Assembly: Sparse Imitation Learning for Text Based Games with Combinatorial Action Spaces
• CREDIT: Coarse-to-Fine Sequence Generation for Dialogue State Tracking
• Differentiable Weighted Finite-State Transducers
• Defending Against Backdoor Attacks in Natural Language Generation
• Debiasing the Cloze Task in Sequential Recommendation with Bidirectional Transformers
• Answers Unite! Unsupervised Metrics for Reinforced Summarization Models
• On the Weaknesses of Reinforcement Learning for Neural Machine Translation
• Joint Training of Candidate Extraction and Answer Selection for Reading Comprehension
• Reinforcement Learning for Weakly Supervised Temporal Grounding of Natural Language in Untrimmed Videos
• Grammatical Error Correction with Neural Reinforcement Learning
• Translating Math Formula Images to LaTeX Sequences Using Deep Neural Networks with Sequence-level Training
• Does Simultaneous Speech Translation need Simultaneous Models?
• Discriminative Adversarial Search for Abstractive Summarization
• Evaluating Transfer Learning for Simplifying GitHub READMEs
• Hard but Robust, Easy but Sensitive: How Encoder and Decoder Perform in Neural Machine Translation
• ACtuAL: Actor-Critic Under Adversarial Learning
• Regularizing Neural Machine Translation by Target-bidirectional Agreement
• Large-scale Pretraining for Neural Machine Translation with Tens of Billions of Sentence Pairs
• Fine-Grained Image Captioning with Global-Local Discriminative Objective
• Beyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter
• Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task
• Improving End-to-End Speech Recognition with Policy Learning
• Discourse-Aware Neural Rewards for Coherent Text Generation
• ReWE: Regressing Word Embeddings for Regularization of Neural Machine Translation Systems
• Machine Translation : From Statistical to modern Deep-learning practices
• APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
• Greedy Search with Probabilistic N-gram Matching for Neural Machine Translation
• A Deep Reinforced Sequence-to-Set Model for Multi-Label Text Classification
• Zero-Resource Neural Machine Translation with Multi-Agent Communication Game
• Dialog State Tracking with Reinforced Data Augmentation
• Tag-less Back-Translation
• ColdGANs: Taming Language GANs with Cautious Sampling Strategies
• Finding It at Another Side: A Viewpoint-Adapted Matching Encoder for Change Captioning
• On the Inference Calibration of Neural Machine Translation
• Proximal Policy Optimization and its Dynamic Version for Sequence Generation
• A New GAN-based End-to-End TTS Training Algorithm
• In-Home Daily-Life Captioning Using Radio Signals
• Teaching Machines to Converse
• A Pilot Study of Domain Adaptation Effect for Neural Abstractive Summarization
• Informative Image Captioning with External Sources of Information
• Non-parallel Voice Conversion System with WaveNet Vocoder and Collapsed Speech Suppression
• Evaluating Rewards for Question Generation Models
• Topo-boundary: A Benchmark Dataset on Topological Road-boundary Detection Using Aerial Images for Autonomous Driving
• Multimedia Generative Script Learning for Task Planning
• SHAPED: Shared-Private Encoder-Decoder for Text Style Adaptation
• A Stable and Effective Learning Strategy for Trainable Greedy Decoding
• Dense Information Flow for Neural Machine Translation
• Reinforcement Learning of Graph Neural Networks for Service Function Chaining
• How Local is the Local Diversity? Reinforcing Sequential Determinantal Point Processes with Dynamic Ground Sets for Supervised Video Summarization
• LeafNATS: An Open-Source Toolkit and Live Demo System for Neural Abstractive Text Summarization
• Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning
• Distilling Knowledge for Search-based Structured Prediction
• Goal-directed Generation of Discrete Structures with Conditional Generative Models
• Towards Amortized Ranking-Critical Training for Collaborative Filtering
• Backprop-Q: Generalized Backpropagation for Stochastic Computation Graphs
• Improving Sequential Determinantal Point Processes for Supervised Video Summarization
• BanditRank: Learning to Rank Using Contextual Bandits
• An Overview of Natural Language State Representation for Reinforcement Learning
• AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks
• ECOL-R: Encouraging Copying in Novel Object Captioning with Reinforcement Learning
• From Credit Assignment to Entropy Regularization: Two New Algorithms for Neural Sequence Prediction
• Text Generation Based on Generative Adversarial Nets with Latent Variable
• Energy-Based Models with Applications to Speech and Language Processing
• Boosting Naturalness of Language in Task-oriented Dialogues via Adversarial Training
• Neural Machine Translation: A Review and Survey
• Paraphrase Generation as Unsupervised Machine Translation
• Sequence-to-Sequence ASR Optimization via Reinforcement Learning
• Improving Reinforcement Learning Based Image Captioning with Natural Language Prior
• Intention Oriented Image Captions with Guiding Objects
• Towards one-shot learning for rare-word translation with external experts
• Promising Accurate Prefix Boosting for sequence-to-sequence ASR
• Mining for meaning: from vision to language through multiple networks consensus
• Paraphrases as Foreign Languages in Multilingual Neural Machine Translation
• Controllable Length Control Neural Encoder-Decoder via Reinforcement Learning
• Learning Domain Adaptation with Model Calibration for Surgical Report Generation in Robotic Surgery
• Transferring Source Style in Non-Parallel Voice Conversion
• A Differentially Private Multi-Output Deep Generative Networks Approach For Activity Diary Synthesis
• Towards Reinforcement Learning for Pivot-based Neural Machine Translation with Non-autoregressive Transformer
• Transfer Reward Learning for Policy Gradient-Based Text Generation
• Seq-SG2SL: Inferring Semantic Layout from Scene Graph Through Sequence to Sequence Learning
• Bayesian Attention Modules
• A Face-to-Face Neural Conversation Model
• SparseGAN: Sparse Generative Adversarial Network for Text Generation
• Context-Dependent Semantic Parsing over Temporally Structured Data
• Differentiable Sampling with Flexible Reference Word Order for Neural Machine Translation
• When Better Eyes Lead to Blindness: A Diagnostic Study of the Information Bottleneck in CNN-LSTM Image Captioning Models
• Enhance Long Text Understanding via Distilled Gist Detector from Abstractive Summarization
• $k$-Neighbor Based Curriculum Sampling for Sequence Prediction
• Stack-VS: Stacked Visual-Semantic Attention for Image Caption Generation
• Boosting Summarization with Normalizing Flows and Aggressive Training
• Autoregressive Knowledge Distillation through Imitation Learning
• Improving Automatic Source Code Summarization via Deep Reinforcement Learning
• Multitasking Inhibits Semantic Drift
• Integrating User and Agent Models: A Deep Task-Oriented Dialogue System
• Ranking sentences from product description & bullets for better search
• Quantity vs. Quality of Monolingual Source Data in Automatic Text Translation: Can It Be Too Little If It Is Too Good?
• Predict and Use Latent Patterns for Short-Text Conversation
• Boosting Image Recognition with Non-differentiable Constraints
• Learn to Talk via Proactive Knowledge Transfer
• The Highs and Lows of Simple Lexical Domain Adaptation Approaches for Neural Machine Translation
• Approximate Distribution Matching for Sequence-to-Sequence Learning
• TAG : Type Auxiliary Guiding for Code Comment Generation