Behavior-Guided Actor-Critic: Improving Exploration via Learning Policy Behavior Representation for Deep Reinforcement Learning
arXiv:2104.04424
Abstract
In this work, we propose Behavior-Guided Actor-Critic (BAC), an off-policy actor-critic deep RL algorithm. BAC mathematically formulates the behavior of the policy through autoencoders by providing an accurate estimation of how frequently each state-action pair was visited while taking into consideration state dynamics that play a crucial role in determining the trajectories produced by the policy. The agent is encouraged to change its behavior consistently towards less-visited state-action pairs while attaining good performance by maximizing the expected discounted sum of rewards, resulting in an efficient exploration of the environment and good exploitation of all high reward regions. One prominent aspect of our approach is that it is applicable to both stochastic and deterministic actors in contrast to maximum entropy deep reinforcement learning algorithms. Results show considerably better performances of BAC when compared to several cutting-edge learning algorithms.
arXiv commentPreprint. Under Review. 9 pages, 3 figures, 1 table
References in corpus (10)
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Addressing Function Approximation Error in Actor-Critic Methods
- Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
- Large-Scale Study of Curiosity-Driven Learning
- Count-Based Exploration with Neural Density Models
- #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
- Unifying Count-Based Exploration and Intrinsic Motivation
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
- Off-Policy Actor-Critic in an Ensemble: Achieving Maximum General Entropy and Effective Environment Exploration in Deep Reinforcement Learning