Maximum a Posteriori Policy Optimisation

Abstract

We introduce a new algorithm for reinforcement learning called Maximum aposteriori Policy Optimisation (MPO) based on coordinate ascent on a relative entropy objective. We show that several existing methods can directly be related to our derivation. We develop two off-policy algorithms and demonstrate that they are competitive with the state-of-the-art in deep reinforcement learning. In particular, for continuous control, our method outperforms existing methods with respect to sample efficiency, premature convergence and robustness to hyperparameter settings while achieving similar or better final performance.

References in corpus (10)

• Benchmarking Deep Reinforcement Learning for Continuous Control
• Emergence of Locomotion Behaviours in Rich Environments
• Generative Adversarial Imitation Learning
• Hindsight Experience Replay
• Path integrals and symmetry breaking for optimal control theory
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates
• Convergent Tree Backup and Retrace with Function Approximation
• Path Integral Guided Policy Search

Cited by in corpus (106)

• Soft Actor-Critic Algorithms and Applications
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Behavior Regularized Offline Reinforcement Learning
• dm_control: Software and Tasks for Continuous Control
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• Critic Regularized Regression
• Efficient Exploration via State Marginal Matching
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• First Order Constrained Optimization in Policy Space
• An empirical investigation of the challenges of real-world reinforcement learning
• Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
• Relative Entropy Regularized Policy Iteration
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• If MaxEnt RL is the Answer, What is the Question?
• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
• VIREL: A Variational Inference Framework for Reinforcement Learning
• Maximum Entropy RL (Provably) Solves Some Robust RL Problems
• Learning to Reach Goals via Iterated Supervised Learning
• Reinforcement Learning via Fenchel-Rockafellar Duality
• Dropout Q-Functions for Doubly Efficient Reinforcement Learning
• Model-Based Offline Planning
• Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
• Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of others
• A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment
• On the model-based stochastic value gradient for continuous reinforcement learning
• Q-Learning for Continuous Actions with Cross-Entropy Guided Policies
• Variational Bayesian Reinforcement Learning with Regret Bounds
• Learning Dexterous Manipulation from Suboptimal Experts
• Driver Assistance Eco-driving and Transmission Control with Deep Reinforcement Learning
• Muesli: Combining Improvements in Policy Optimization
• Connecting the Dots Between MLE and RL for Sequence Prediction
• Simplifying Deep Reinforcement Learning via Self-Supervision
• Making Sense of Reinforcement Learning and Probabilistic Inference
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
• Residual Reinforcement Learning from Demonstrations
• Human-in-the-Loop Imitation Learning using Remote Teleoperation
• Divergence-Augmented Policy Optimization
• Improving GAN Training with Probability Ratio Clipping and Sample Reweighting
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
• Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences
• Disentangled Skill Embeddings for Reinforcement Learning
• Safety Filtering for Reinforcement Learning-based Adaptive Cruise Control
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
• AndroidEnv: A Reinforcement Learning Platform for Android
• Adaptive Approximate Policy Iteration
• Data-efficient Hindsight Off-policy Option Learning
• Stable Policy Optimization via Off-Policy Divergence Regularization
• Learning to Communicate Implicitly By Actions
• Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration
• Local Search for Policy Iteration in Continuous Control
• Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking
• Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies
• Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
• Augmenting learning using symmetry in a biologically-inspired domain
• Plan2Vec: Unsupervised Representation Learning by Latent Plans
• Colored Noise in PPO: Improved Exploration and Performance through Correlated Action Sampling
• Safe Reinforcement Learning for an Energy-Efficient Driver Assistance System
• RLSS: A Deep Reinforcement Learning Algorithm for Sequential Scene Generation
• Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
• ${\alpha }^{\alpha }$-Rank: Practically Scaling $\alpha$-Rank through Stochastic Optimisation
• An operator view of policy gradient methods
• Learning Dynamics Models for Model Predictive Agents
• Taylor Expansion Policy Optimization
• Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification
• Quinoa: a Q-function You Infer Normalized Over Actions
• Iterative Amortized Policy Optimization
• Mutual-Information Regularization in Markov Decision Processes and Actor-Critic Learning
• Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning
• Residual Policy Learning for Powertrain Control
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Policy Search by Target Distribution Learning for Continuous Control
• Simple Sensor Intentions for Exploration
• Control as Hybrid Inference
• Market-making with reinforcement-learning (SAC)
• A Relation Analysis of Markov Decision Process Frameworks
• Shared learning of powertrain control policies for vehicle fleets
• Exploration by Maximizing Rényi Entropy for Reward-Free RL Framework
• Direct Policy Gradients: Direct Optimization of Policies in Discrete Action Spaces
• Snowflake: Scaling GNNs to High-Dimensional Continuous Control via Parameter Freezing
• Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learning
• Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Control
• Implicitly Regularized RL with Implicit Q-Values
• Actor-critic is implicitly biased towards high entropy optimal policies
• Manipulator-Independent Representations for Visual Imitation
• Reinforcement Learning for Vision-based Object Manipulation with Non-parametric Policy and Action Primitives
• Logistic Q-Learning
• "What, not how": Solving an under-actuated insertion task from scratch
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Near Optimal Policy Optimization via REPS
• Self-Paced Deep Reinforcement Learning
• Reward is enough for convex MDPs
• A Quadratic Actor Network for Model-Free Reinforcement Learning
• Bootstrapped Meta-Learning
• A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning
• Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learning
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning
• Ranking Policy Gradient
• Integration of Imitation Learning using GAIL and Reinforcement Learning using Task-achievement Rewards via Probabilistic Graphical Model
• Towards an Understanding of Default Policies in Multitask Policy Optimization