Reinforcement Learning with Random Delays
arXiv:2010.02966
Abstract
Action and observation delays commonly occur in many Reinforcement Learning applications, such as remote control scenarios. We study the anatomy of randomly delayed environments, and show that partially resampling trajectory fragments in hindsight allows for off-policy multi-step value estimation. We apply this principle to derive Delay-Correcting Actor-Critic (DCAC), an algorithm based on Soft Actor-Critic with significantly better performance in environments with delays. This is shown theoretically and also demonstrated practically on a delay-augmented version of the MuJoCo continuous control benchmark.
arXiv commentICLR 2021
References in corpus (7)
- Adam: A Method for Stochastic Optimization
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Addressing Function Approximation Error in Actor-Critic Methods
- Soft Actor-Critic Algorithms and Applications
- At Human Speed: Deep Reinforcement Learning with Action Delay
- Real-Time Reinforcement Learning
- Thinking While Moving: Deep Reinforcement Learning with Concurrent Control