Trust-PCL: An Off-Policy Trust Region Method for Continuous Control
arXiv:1707.01891
Abstract
Trust region methods, such as TRPO, are often used to stabilize policy optimization algorithms in reinforcement learning (RL). While current trust region strategies are effective for continuous control, they typically require a prohibitively large amount of on-policy interaction with the environment. To address this problem, we propose an off-policy trust region method, Trust-PCL. The algorithm is the result of observing that the optimal policy and state values of a maximum reward objective with a relative-entropy regularizer satisfy a set of multi-step pathwise consistencies along any path. Thus, Trust-PCL is able to maintain optimization stability while exploiting off-policy data to improve sample efficiency. When evaluated on a number of continuous control tasks, Trust-PCL improves the solution quality and sample efficiency of TRPO.
arXiv commentICLR 2018
References in corpus (11)
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- Benchmarking Deep Reinforcement Learning for Continuous Control
- Emergence of Locomotion Behaviours in Rich Environments
- VIME: Variational Information Maximizing Exploration
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Bridging the Gap Between Value and Policy Based Reinforcement Learning
- Sample Efficient Actor-Critic with Experience Replay
- Equivalence Between Policy Gradients and Soft Q-Learning
- Discrete Sequential Prediction of Continuous Actions for Deep RL
- Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning
Cited by in corpus (14)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Soft Actor-Critic Algorithms and Applications
- Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past
- TD-Regularized Actor-Critic Methods
- Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration
- Continuous-action Reinforcement Learning for Playing Racing Games: Comparing SPG to PPO
- Hindsight Trust Region Policy Optimization
- Direct and indirect reinforcement learning
- Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation
- A Max-Min Entropy Framework for Reinforcement Learning
- Generalized Off-Policy Actor-Critic
- Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learning
- A short variational proof of equivalence between policy gradients and soft Q learning
- Measuring Progress in Deep Reinforcement Learning Sample Efficiency