Qualitative Measurements of Policy Discrepancy for Return-Based Deep Q-Network

Abstract

The deep Q-network (DQN) and return-based reinforcement learning are two promising algorithms proposed in recent years. DQN brings advances to complex sequential decision problems, while return-based algorithms have advantages in making use of sample trajectories. In this paper, we propose a general framework to combine DQN and most of the return-based reinforcement learning algorithms, named R-DQN. We show the performance of traditional DQN can be improved effectively by introducing return-based reinforcement learning. In order to further improve the R-DQN, we design a strategy with two measurements which can qualitatively measure the policy discrepancy. Moreover, we give the two measurements' bounds in the proposed R-DQN framework. We show that algorithms with our strategy can accurately express the trace coefficient and achieve a better approximation to return. The experiments, conducted on several representative tasks from the OpenAI Gym library, validate the effectiveness of the proposed measurements. The results also show that the algorithms with our strategy outperform the state-of-the-art methods.

References in corpus (9)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Prioritized Experience Replay
• Dueling Network Architectures for Deep Reinforcement Learning
• OpenAI Gym
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning
• The Reactor: A fast and sample-efficient Actor-Critic agent for Reinforcement Learning
• Q($\lambda$) with Off-Policy Corrections

Cited by in corpus (4)

• Evolutionary learning of interpretable decision trees
• Optimize Neural Fictitious Self-Play in Regret Minimization Thinking
• The Chef's Hat Simulation Environment for Reinforcement-Learning-Based Agents
• Continuous Deep Q-Learning with Simulator for Stabilization of Uncertain Discrete-Time Systems