Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog

Abstract

Most deep reinforcement learning (RL) systems are not able to learn effectively from off-policy data, especially if they cannot explore online in the environment. These are critical shortcomings for applying RL to real-world problems where collecting data is expensive, and models must be tested offline before being deployed to interact with the environment -- e.g. systems that learn from human interaction. Thus, we develop a novel class of off-policy batch RL algorithms, which are able to effectively learn offline, without exploring, from a fixed batch of human interaction data. We leverage models pre-trained on data as a strong prior, and use KL-control to penalize divergence from this prior during RL training. We also use dropout-based uncertainty estimates to lower bound the target Q-values as a more efficient alternative to Double Q-Learning. The algorithms are tested on the problem of open-domain dialog generation -- a challenging reinforcement learning problem with a 20,000-dimensional action space. Using our Way Off-Policy algorithm, we can extract multiple different reward functions post-hoc from collected human interaction data, and learn effectively from all of these. We test the real-world generalization of these systems by deploying them live to converse with humans in an open-domain setting, and demonstrate that our algorithm achieves significant improvements over prior methods in off-policy batch RL.

References in corpus (15)

• Distilling the Knowledge in a Neural Network
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Reinforcement Learning with Deep Energy-Based Policies
• Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs
• Optimal control as a graphical model inference problem
• Uncertainty-Aware Reinforcement Learning for Collision Avoidance
• Maximum a Posteriori Policy Optimisation
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• A Deep Reinforcement Learning Chatbot
• Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems
• Off-Policy Actor-Critic
• Dialogue Learning With Human-In-The-Loop
• Off-Policy Policy Gradient with State Distribution Correction
• A Hierarchical Latent Structure for Variational Conversation Modeling

Cited by in corpus (59)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Fine-Tuning Language Models from Human Preferences
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• A Minimalist Approach to Offline Reinforcement Learning
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• MOReL : Model-Based Offline Reinforcement Learning
• Critic Regularized Regression
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Recursively Summarizing Books with Human Feedback
• COMBO: Conservative Offline Model-Based Policy Optimization
• An empirical investigation of the challenges of real-world reinforcement learning
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• Learning to summarize from human feedback
• A Distributional Approach to Controlled Text Generation
• BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning
• Offline Reinforcement Learning with Fisher Divergence Critic Regularization
• Neural Language Generation: Formulation, Methods, and Evaluation
• The Importance of Pessimism in Fixed-Dataset Policy Optimization
• Representation Matters: Offline Pretraining for Sequential Decision Making
• Offline Reinforcement Learning with Reverse Model-based Imagination
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
• Value Penalized Q-Learning for Recommender Systems
• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning
• Offline RL Without Off-Policy Evaluation
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• Risk-Averse Offline Reinforcement Learning
• Behavior Priors for Efficient Reinforcement Learning
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• PLAS: Latent Action Space for Offline Reinforcement Learning
• Offline Meta-Reinforcement Learning with Advantage Weighting
• PyTester: Deep Reinforcement Learning for Text-to-Testcase Generation
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• Causal-aware Safe Policy Improvement for Task-oriented dialogue
• Medical Dead-ends and Learning to Identify High-risk States and Treatments
• Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendation
• EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• Text Generation by Learning from Demonstrations
• Offline Reinforcement Learning with Pseudometric Learning
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
• Document-editing Assistants and Model-based Reinforcement Learning as a Path to Conversational AI
• Regularized Behavior Value Estimation
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Offline Decentralized Multi-Agent Reinforcement Learning
• On the Optimality of Batch Policy Optimization Algorithms
• Reducing Conservativeness Oriented Offline Reinforcement Learning
• Discovering an Aid Policy to Minimize Student Evasion Using Offline Reinforcement Learning
• Offline Inverse Reinforcement Learning
• Learning Natural Language Generation from Scratch
• Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach
• Off-Policy Self-Critical Training for Transformer in Visual Paragraph Generation
• Enhancing Reinforcement Learning Through Guided Search