Making Deep Q-learning methods robust to time discretization

Abstract

Despite remarkable successes, Deep Reinforcement Learning (DRL) is not robust to hyperparameterization, implementation details, or small environment changes (Henderson et al. 2017, Zhang et al. 2018). Overcoming such sensitivity is key to making DRL applicable to real world problems. In this paper, we identify sensitivity to time discretization in near continuous-time environments as a critical factor; this covers, e.g., changing the number of frames per second, or the action frequency of the controller. Empirically, we find that Q-learning-based approaches such as Deep Q- learning (Mnih et al., 2015) and Deep Deterministic Policy Gradient (Lillicrap et al., 2015) collapse with small time steps. Formally, we prove that Q-learning does not exist in continuous time. We detail a principled way to build an off-policy RL algorithm that yields similar performances over a wide range of time discretizations, and confirm this robustness empirically.

References in corpus (5)

• Trust Region Policy Optimization
• Dueling Network Architectures for Deep Reinforcement Learning
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Learning Dexterous In-Hand Manipulation
• Natural Environment Benchmarks for Reinforcement Learning

Cited by in corpus (3)

• Policy Iterations for Reinforcement Learning Problems in Continuous Time and Space -- Fundamental Theory and Methods
• Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs
• Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon