V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control

Abstract

Some of the most successful applications of deep reinforcement learning to challenging domains in discrete and continuous control have used policy gradient methods in the on-policy setting. However, policy gradients can suffer from large variance that may limit performance, and in practice require carefully tuned entropy regularization to prevent policy collapse. As an alternative to policy gradient algorithms, we introduce V-MPO, an on-policy adaptation of Maximum a Posteriori Policy Optimization (MPO) that performs policy iteration based on a learned state-value function. We show that V-MPO surpasses previously reported scores for both the Atari-57 and DMLab-30 benchmark suites in the multi-task setting, and does so reliably without importance weighting, entropy regularization, or population-based tuning of hyperparameters. On individual DMLab and Atari levels, the proposed algorithm can achieve scores that are substantially higher than has previously been reported. V-MPO is also applicable to problems with high-dimensional, continuous action spaces, which we demonstrate in the context of learning to control simulated humanoids with 22 degrees of freedom from full state observations and 56 degrees of freedom from pixel observations, as well as example OpenAI Gym tasks where V-MPO achieves substantially higher asymptotic scores than previously reported.

arXiv comment* equal contribution

References in corpus (14)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Asynchronous Methods for Deep Reinforcement Learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Neural probabilistic motor primitives for humanoid control
• Relative Entropy Regularized Policy Iteration
• Kickstarting Deep Reinforcement Learning
• Multi-task Deep Reinforcement Learning with PopArt
• TF-Replicator: Distributed Machine Learning for Researchers

Cited by in corpus (19)

• Stabilizing Transformers for Reinforcement Learning
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• Phasic Policy Gradient
• Reinforcement Learning via Fenchel-Rockafellar Duality
• Revisiting Design Choices in Proximal Policy Optimization
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
• Deep neuroethology of a virtual rodent
• Local Search for Policy Iteration in Continuous Control
• Using Probabilistic Movement Primitives in Analyzing Human Motion Difference under Transcranial Current Stimulation
• Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
• Taylor Expansion Policy Optimization
• My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control
• Zeroth-Order Supervised Policy Improvement
• Optimization Issues in KL-Constrained Approximate Policy Iteration
• Statistical discrimination in learning agents
• Stabilizing Transformer-Based Action Sequence Generation For Q-Learning
• Logistic Q-Learning
• Towards an Understanding of Default Policies in Multitask Policy Optimization
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning