Iterative Amortized Policy Optimization
arXiv:2010.10670
Abstract
Policy networks are a central feature of deep reinforcement learning (RL) algorithms for continuous control, enabling the estimation and sampling of high-value actions. From the variational inference perspective on RL, policy networks, when used with entropy or KL regularization, are a form of \textit{amortized optimization}, optimizing network parameters rather than the policy distributions directly. However, \textit{direct} amortized mappings can yield suboptimal policy estimates and restricted distributions, limiting performance and exploration. Given this perspective, we consider the more flexible class of \textit{iterative} amortized optimizers. We demonstrate that the resulting technique, iterative amortized policy optimization, yields performance improvements over direct amortization on benchmark continuous control tasks.
arXiv commentAdvances in Neural Processing Systems (NeurIPS) 2021
References in corpus (27)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Trust Region Policy Optimization
- Soft Actor-Critic Algorithms and Applications
- Training Very Deep Networks
- QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
- Conservative Q-Learning for Offline Reinforcement Learning
- Reinforcement Learning with Deep Energy-Based Policies
- Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
- Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
- Adaptive Computation Time for Recurrent Neural Networks
- Learning Continuous Control Policies by Stochastic Value Gradients
- Learning Latent Dynamics for Planning from Pixels
- Multi-Object Representation Learning with Iterative Variational Inference
- Maximum a Posteriori Policy Optimisation
- Neural Variational Inference and Learning in Belief Networks
- A Method for Using Belief Networks as Influence Diagrams
- A Game Theoretic Framework for Model Based Reinforcement Learning
- Model-Augmented Actor-Critic: Backpropagating through Paths
- Better Exploration with Optimistic Actor-Critic
- Model-Based Planning with Discrete and Continuous Actions
- Boosting Trust Region Policy Optimization by Normalizing Flows Policy
- Safe end-to-end imitation learning for model predictive control
- Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
- A General Method for Amortizing Variational Filtering