Distributed Soft Actor-Critic with Multivariate Reward Representation and Knowledge Distillation
arXiv:1911.13056
Abstract
In this paper, we describe NeurIPS 2019 Learning to Move - Walk Around challenge physics-based environment and present our solution to this competition which scored 1303.727 mean reward points and took 3rd place. Our method combines recent advances from both continuous- and discrete-action space reinforcement learning, such as Soft Actor-Critic and Recurrent Experience Replay in Distributed Reinforcement Learning. We trained our agent in two stages: to move somewhere at the first stage and to follow the target velocity field at the second stage. We also introduce novel Q-function split technique, which we believe facilitates the task of training an agent, allows critic pretraining and reusing it for solving harder problems, and mitigate reward shaping design efforts.
arXiv comment9 pages, 5 figures
References in corpus (9)
- Adam: A Method for Stochastic Optimization
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Addressing Function Approximation Error in Actor-Critic Methods
- Prioritized Experience Replay
- Soft Actor-Critic Algorithms and Applications
- Distributed Prioritized Experience Replay
- Distributed Distributional Deterministic Policy Gradients
- Latent Space Policies for Hierarchical Reinforcement Learning