Bridging the Gap Between Value and Policy Based Reinforcement Learning

Abstract

We establish a new connection between value and policy based reinforcement learning (RL) based on a relationship between softmax temporal value consistency and policy optimality under entropy regularization. Specifically, we show that softmax consistent action values correspond to optimal entropy regularized policy probabilities along any action sequence, regardless of provenance. From this observation, we develop a new RL algorithm, Path Consistency Learning (PCL), that minimizes a notion of soft consistency error along multi-step action sequences extracted from both on- and off-policy traces. We examine the behavior of PCL in different scenarios and show that PCL can be interpreted as generalizing both actor-critic and Q-learning algorithms. We subsequently deepen the relationship by showing how a single model can be used to represent both a policy and the corresponding softmax state values, eliminating the need for a separate critic. The experimental evaluation demonstrates that PCL significantly outperforms strong actor-critic and Q-learning baselines across several benchmarks.

NIPS 2017

Cited by in corpus (112)

• A Brief Survey of Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Soft Actor-Critic Algorithms and Applications
• Deep Reinforcement Learning for Cyber Security
• Deep Reinforcement Learning: An Overview
• Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors
• Distral: Robust Multitask Reinforcement Learning
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• A survey on text generation using generative adversarial networks
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Reinforcement Learning from Imperfect Demonstrations
• On the Global Convergence Rates of Softmax Policy Gradient Methods
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Learning Montezuma's Revenge from a Single Demonstration
• Latent Space Policies for Hierarchical Reinforcement Learning
• Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition
• A reinforcement learning approach to rare trajectory sampling
• Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
• Relative Entropy Regularized Policy Iteration
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization
• Value Propagation for Decentralized Networked Deep Multi-agent Reinforcement Learning
• If MaxEnt RL is the Answer, What is the Question?
• Maximum Entropy RL (Provably) Solves Some Robust RL Problems
• Trust-PCL: An Off-Policy Trust Region Method for Continuous Control
• Toward Diverse Text Generation with Inverse Reinforcement Learning
• An Adaptive Clipping Approach for Proximal Policy Optimization
• Mirror Descent Policy Optimization
• Recall Traces: Backtracking Models for Efficient Reinforcement Learning
• Exploration versus exploitation in reinforcement learning: a stochastic control approach
• Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learning
• Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
• Distributionally Robust Reinforcement Learning
• An Information-Theoretic Optimality Principle for Deep Reinforcement Learning
• A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment
• Behavior Priors for Efficient Reinforcement Learning
• Variational Bayesian Reinforcement Learning with Regret Bounds
• Efficient (Soft) Q-Learning for Text Generation with Limited Good Data
• Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform Sampling
• A Tutorial on Sparse Gaussian Processes and Variational Inference
• MADE: Exploration via Maximizing Deviation from Explored Regions
• Self-Imitation Learning via Generalized Lower Bound Q-learning
• Divergence-Augmented Policy Optimization
• Smoothed Action Value Functions for Learning Gaussian Policies
• Batch-Constrained Reinforcement Learning for Dynamic Distribution Network Reconfiguration
• Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration
• Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences
• Direct and indirect reinforcement learning
• Equivalence Between Wasserstein and Value-Aware Loss for Model-based Reinforcement Learning
• Path Consistency Learning in Tsallis Entropy Regularized MDPs
• A general sample complexity analysis of vanilla policy gradient
• Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality
• Provable Fictitious Play for General Mean-Field Games
• Effective Exploration for Deep Reinforcement Learning via Bootstrapped Q-Ensembles under Tsallis Entropy Regularization
• EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL
• Situated GAIL: Multitask imitation using task-conditioned adversarial inverse reinforcement learning
• VFunc: a Deep Generative Model for Functions
• PixelRL: Fully Convolutional Network with Reinforcement Learning for Image Processing
• Scalable Online Disease Diagnosis via Multi-Model-Fused Actor-Critic Reinforcement Learning
• Implicit Policy for Reinforcement Learning
• Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning
• Can Temporal-Difference and Q-Learning Learn Representation? A Mean-Field Theory
• A Max-Min Entropy Framework for Reinforcement Learning
• Convergent and Efficient Deep Q Network Algorithm
• An operator view of policy gradient methods
• Regularity and stability of feedback relaxed controls
• Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation
• Off-Policy Actor-Critic in an Ensemble: Achieving Maximum General Entropy and Effective Environment Exploration in Deep Reinforcement Learning
• Consensus Multi-Agent Reinforcement Learning for Volt-VAR Control in Power Distribution Networks
• Only Relevant Information Matters: Filtering Out Noisy Samples to Boost RL
• Generalized Off-Policy Actor-Critic
• Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learning
• Learning and Management for Internet-of-Things: Accounting for Adaptivity and Scalability
• P3O: Policy-on Policy-off Policy Optimization
• A Dual Approach to Constrained Markov Decision Processes with Entropy Regularization
• Review, Analysis and Design of a Comprehensive Deep Reinforcement Learning Framework
• Pretrain Soft Q-Learning with Imperfect Demonstrations
• Control as Hybrid Inference
• From Credit Assignment to Entropy Regularization: Two New Algorithms for Neural Sequence Prediction
• MAMRL: Exploiting Multi-agent Meta Reinforcement Learning in WAN Traffic Engineering
• A Bandit Framework for Optimal Selection of Reinforcement Learning Agents
• Mutual-Information Regularization in Markov Decision Processes and Actor-Critic Learning
• Sample-Efficient Reinforcement Learning with Maximum Entropy Mellowmax Episodic Control
• Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learning
• Merging Deterministic Policy Gradient Estimations with Varied Bias-Variance Tradeoff for Effective Deep Reinforcement Learning
• Long Short-Term Memory with Dynamic Skip Connections
• Divergence-Regularized Multi-Agent Actor-Critic
• OPAC: Opportunistic Actor-Critic
• A Convergence Result for Regularized Actor-Critic Methods
• Exploration by Maximizing Rényi Entropy for Reward-Free RL Framework
• On the Convergence of Approximate and Regularized Policy Iteration Schemes
• Reinforcement Learning as Iterative and Amortised Inference
• Advanced Policies: A First-Principles Path from Policy Gradient to Q-Learning
• Near Optimal Policy Optimization via REPS
• An Entropy Regularization Free Mechanism for Policy-based Reinforcement Learning
• Implicitly Regularized RL with Implicit Q-Values
• ISL: A novel approach for deep exploration
• Compositional planning in Markov decision processes: Temporal abstraction meets generalized logic composition
• Will it Blend? Composing Value Functions in Reinforcement Learning
• Guide Actor-Critic for Continuous Control
• Parameterized MDPs and Reinforcement Learning Problems -- A Maximum Entropy Principle Based Framework
• Planning and Learning Using Adaptive Entropy Tree Search
• Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learning
• Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatch
• Count-Based Temperature Scheduling for Maximum Entropy Reinforcement Learning
• A new soft computing method for integration of expert's knowledge in reinforcement learn-ing problems
• Cautious Actor-Critic
• Mapping Language to Programs using Multiple Reward Components with Inverse Reinforcement Learning
• Weighted Entropy Modification for Soft Actor-Critic
• Transfer Learning by Modeling a Distribution over Policies
• Ranking Policy Gradient