Evolution of Q Values for Deep Q Learning in Stable Baselines

Abstract

We investigate the evolution of the Q values for the implementation of Deep Q Learning (DQL) in the Stable Baselines library. Stable Baselines incorporates the latest Reinforcement Learning techniques and achieves superhuman performance in many game environments. However, for some simple non-game environments, the DQL in Stable Baselines can struggle to find the correct actions. In this paper we aim to understand the types of environment where this suboptimal behavior can happen, and also investigate the corresponding evolution of the Q values for individual states. We compare a smart TrafficLight environment (where performance is poor) with the AI Gym FrozenLake environment (where performance is perfect). We observe that DQL struggles with TrafficLight because actions are reversible and hence the Q values in a given state are closer than in FrozenLake. We then investigate the evolution of the Q values using a recent decomposition technique of Achiam et al.. We observe that for TrafficLight, the function approximation error and the complex relationships between the states lead to a situation where some Q values meander far from optimal.

References in corpus (11)

• Playing Atari with Deep Reinforcement Learning
• Addressing Function Approximation Error in Actor-Critic Methods
• Dueling Network Architectures for Deep Reinforcement Learning
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning
• A Theoretical Analysis of Deep Q-Learning
• Deep Reinforcement Learning and the Deadly Triad
• Towards Characterizing Divergence in Deep Q-Learning
• Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
• Diagnosing Bottlenecks in Deep Q-learning Algorithms