GRAC: Self-Guided and Self-Regularized Actor-Critic
arXiv:2009.08973
Abstract
Deep reinforcement learning (DRL) algorithms have successfully been demonstrated on a range of challenging decision making and control tasks. One dominant component of recent deep reinforcement learning algorithms is the target network which mitigates the divergence when learning the Q function. However, target networks can slow down the learning process due to delayed function updates. Our main contribution in this work is a self-regularized TD-learning method to address divergence without requiring a target network. Additionally, we propose a self-guided policy improvement method by combining policy-gradient with zero-order optimization to search for actions associated with higher Q-values in a broad neighborhood. This makes learning more robust to local noise in the Q function approximation and guides the updates of our actor network. Taken together, these components define GRAC, a novel self-guided and self-regularized actor critic algorithm. We evaluate GRAC on the suite of OpenAI gym tasks, achieving or outperforming state of the art in every environment tested.
References in corpus (12)
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Trust Region Policy Optimization
- Addressing Function Approximation Error in Actor-Critic Methods
- QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
- Deep Reinforcement Learning and the Deadly Triad
- CEM-RL: Combining evolutionary and gradient-based methods for policy search
- Towards Characterizing Divergence in Deep Q-Learning
- Q-Learning for Continuous Actions with Cross-Entropy Guided Policies
- Learning Probabilistic Multi-Modal Actor Models for Vision-Based Robotic Grasping