Recurrent Off-policy Baselines for Memory-based Continuous Control

Abstract

When the environment is partially observable (PO), a deep reinforcement learning (RL) agent must learn a suitable temporal representation of the entire history in addition to a strategy to control. This problem is not novel, and there have been model-free and model-based algorithms proposed for this problem. However, inspired by recent success in model-free image-based RL, we noticed the absence of a model-free baseline for history-based RL that (1) uses full history and (2) incorporates recent advances in off-policy continuous control. Therefore, we implement recurrent versions of DDPG, TD3, and SAC (RDPG, RTD3, and RSAC) in this work, evaluate them on short-term and long-term PO domains, and investigate key design choices. Our experiments show that RDPG and RTD3 can surprisingly fail on some domains and that RSAC is the most reliable, reaching near-optimal performance on nearly all domains. However, one task that requires systematic exploration still proved to be difficult, even for RSAC. These results show that model-free RL can learn good temporal representation using only reward signals; the primary difficulty seems to be computational cost and exploration. To facilitate future research, we have made our PyTorch implementation publicly available at https://github.com/zhihanyang2022/off-policy-continuous-control.

References in corpus (12)

• Continuous control with deep reinforcement learning
• LSTM: A Search Space Odyssey
• Soft Actor-Critic Algorithms and Applications
• Reinforcement Learning with Deep Energy-Based Policies
• Learning Latent Dynamics for Planning from Pixels
• Memory-based control with recurrent neural networks
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Tianshou: a Highly Modularized Deep Reinforcement Learning Library
• Playing FPS Games with Deep Reinforcement Learning
• On Improving Deep Reinforcement Learning for POMDPs
• Discriminative Particle Filter Reinforcement Learning for Complex Partial Observations
• Belief-Grounded Networks for Accelerated Robot Learning under Partial Observability