Deep Reinforcement Learning with Weighted Q-Learning

Abstract

Reinforcement learning algorithms based on Q-learning are driving Deep Reinforcement Learning (DRL) research towards solving complex problems and achieving super-human performance on many of them. Nevertheless, Q-Learning is known to be positively biased since it learns by using the maximum over noisy estimates of expected values. Systematic overestimation of the action values coupled with the inherently high variance of DRL methods can lead to incrementally accumulate errors, causing learning algorithms to diverge. Ideally, we would like DRL agents to take into account their own uncertainty about the optimality of each action, and be able to exploit it to make more informed estimations of the expected return. In this regard, Weighted Q-Learning (WQL) effectively reduces bias and shows remarkable results in stochastic environments. WQL uses a weighted sum of the estimated action values, where the weights correspond to the probability of each action value being the maximum; however, the computation of these probabilities is only practical in the tabular setting. In this work, we provide methodological advances to benefit from the WQL properties in DRL, by using neural networks trained with Dropout as an effective approximation of deep Gaussian processes. In particular, we adopt the Concrete Dropout variant to obtain calibrated estimates of epistemic uncertainty in DRL. The estimator, then, is obtained by taking several stochastic forward passes through the action-value network and computing the weights in a Monte Carlo fashion. Such weights are Bayesian estimates of the probability of each action value corresponding to the maximum w.r.t. a posterior probability distribution estimated by Dropout. We show how our novel Deep Weighted Q-Learning algorithm reduces the bias w.r.t. relevant baselines and provides empirical evidence of its advantages on representative benchmarks.

RLDM 2022. For a complete discussion and additional results, check our JMLR paper at https://www.jmlr.org/papers/v22/20-633.html

References in corpus (12)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Dueling Network Architectures for Deep Reinforcement Learning
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Uncertainty-Aware Reinforcement Learning for Collision Avoidance
• Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• Generalization and Regularization in DQN
• MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments
• MushroomRL: Simplifying Reinforcement Learning Research
• Calibrated Model-Based Deep Reinforcement Learning
• Estimating the Maximum Expected Value: An Analysis of (Nested) Cross Validation and the Maximum Sample Average
• Deep Contextual Multi-armed Bandits