Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning

Abstract

Instability and variability of Deep Reinforcement Learning (DRL) algorithms tend to adversely affect their performance. Averaged-DQN is a simple extension to the DQN algorithm, based on averaging previously learned Q-values estimates, which leads to a more stable training procedure and improved performance by reducing approximation error variance in the target values. To understand the effect of the algorithm, we examine the source of value function estimation errors and provide an analytical comparison within a simplified model. We further present experiments on the Arcade Learning Environment benchmark that demonstrate significantly improved stability and performance due to the proposed extension.

References in corpus (3)

• Playing Atari with Deep Reinforcement Learning
• Prioritized Experience Replay
• Dueling Network Architectures for Deep Reinforcement Learning

Cited by in corpus (51)

• Addressing Function Approximation Error in Actor-Critic Methods
• Deep Reinforcement Learning: An Overview
• A Survey of Deep Reinforcement Learning in Video Games
• An Optimistic Perspective on Offline Reinforcement Learning
• Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics
• Softmax Deep Double Deterministic Policy Gradients
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• Maxmin Q-learning: Controlling the Estimation Bias of Q-learning
• Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
• Qualitative Measurements of Policy Discrepancy for Return-Based Deep Q-Network
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
• Generalizable Episodic Memory for Deep Reinforcement Learning
• Dropout Q-Functions for Doubly Efficient Reinforcement Learning
• Distributionally Robust Reinforcement Learning
• The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models
• Regularized Softmax Deep Multi-Agent $Q$-Learning
• A Deep Ensemble Multi-Agent Reinforcement Learning Approach for Air Traffic Control
• Ctrl-Z: Recovering from Instability in Reinforcement Learning
• On the Estimation Bias in Double Q-Learning
• Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
• Self-Imitation Learning via Generalized Lower Bound Q-learning
• Interference and Generalization in Temporal Difference Learning
• The Effect of Multi-step Methods on Overestimation in Deep Reinforcement Learning
• Fitted Q-Learning for Relational Domains
• Convex Q-Learning, Part 1: Deterministic Optimal Control
• Variance Reduction for Deep Q-Learning using Stochastic Recursive Gradient
• Qgraph-bounded Q-learning: Stabilizing Model-Free Off-Policy Deep Reinforcement Learning
• The Mean-Squared Error of Double Q-Learning
• The Actor-Advisor: Policy Gradient With Off-Policy Advice
• Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance
• Stable deep reinforcement learning method by predicting uncertainty in rewards as a subtask
• Detect or Track: Towards Cost-Effective Video Object Detection/Tracking
• Merging Deterministic Policy Gradient Estimations with Varied Bias-Variance Tradeoff for Effective Deep Reinforcement Learning
• Decorrelated Double Q-learning
• On The Effect of Auxiliary Tasks on Representation Dynamics
• Stabilizing Q Learning Via Soft Mellowmax Operator
• ConQUR: Mitigating Delusional Bias in Deep Q-learning
• A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets
• Interpretable performance analysis towards offline reinforcement learning: A dataset perspective
• Action Candidate Based Clipped Double Q-learning for Discrete and Continuous Action Tasks
• Robust Risk-Sensitive Reinforcement Learning Agents for Trading Markets
• Deep Reinforcement Learning with Weighted Q-Learning
• Ensemble Bootstrapping for Q-Learning
• Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning
• In Hindsight: A Smooth Reward for Steady Exploration
• Evolution of Q Values for Deep Q Learning in Stable Baselines
• Some Limit Properties of Markov Chains Induced by Stochastic Recursive Algorithms
• Self-correcting Q-Learning
• Maximum Entropy Reinforcement Learning with Mixture Policies
• Coordinate-wise Control Variates for Deep Policy Gradients
• Cross Learning in Deep Q-Networks