MLE-guided parameter search for task loss minimization in neural sequence modeling
arXiv:2006.03158
Abstract
Neural autoregressive sequence models are used to generate sequences in a variety of natural language processing (NLP) tasks, where they are evaluated according to sequence-level task losses. These models are typically trained with maximum likelihood estimation, which ignores the task loss, yet empirically performs well as a surrogate objective. Typical approaches to directly optimizing the task loss such as policy gradient and minimum risk training are based around sampling in the sequence space to obtain candidate update directions that are scored based on the loss of a single sequence. In this paper, we develop an alternative method based on random search in the parameter space that leverages access to the maximum likelihood gradient. We propose maximum likelihood guided parameter search (MGS), which samples from a distribution over update directions that is a mixture of random search around the current parameters and around the maximum likelihood gradient, with each direction weighted by its improvement in the task loss. MGS shifts sampling to the parameter space, and scores candidates using losses that are pooled from multiple sequences. Our experiments show that MGS is capable of optimizing sequence-level losses, with substantial reductions in repetition and non-termination in sequence completion, and similar improvements to those of minimum risk training in machine translation.
References in corpus (22)
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- Language Models are Few-Shot Learners
- A Neural Conversational Model
- A Deep Reinforced Model for Abstractive Summarization
- The Curious Case of Neural Text Degeneration
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Pointer Sentinel Mixture Models
- Improved Image Captioning via Policy Gradient optimization of SPIDEr
- Fine-Tuning Language Models from Human Preferences
- Deep Reinforcement Learning that Matters
- An Actor-Critic Algorithm for Sequence Prediction
- Simple random search provides a competitive approach to reinforcement learning
- CEM-RL: Combining evolutionary and gradient-based methods for policy search
- Reward Augmented Maximum Likelihood for Neural Structured Prediction
- Guided evolutionary strategies: Augmenting random search with surrogate gradients
- Contrasting Exploration in Parameter and Action Space: A Zeroth-Order Optimization Perspective
- Residual Energy-Based Models for Text Generation
- Injecting External Solutions Into CMA-ES
- Consistency of a Recurrent Language Model With Respect to Incomplete Decoding
- Non-Differentiable Supervised Learning with Evolution Strategies and Hybrid Methods
- Learning to Guide Random Search
- On the Discrepancy between Density Estimation and Sequence Generation