Munchausen Reinforcement Learning

Abstract

Bootstrapping is a core mechanism in Reinforcement Learning (RL). Most algorithms, based on temporal differences, replace the true value of a transiting state by their current estimate of this value. Yet, another estimate could be leveraged to bootstrap RL: the current policy. Our core contribution stands in a very simple idea: adding the scaled log-policy to the immediate reward. We show that slightly modifying Deep Q-Network (DQN) in that way provides an agent that is competitive with distributional methods on Atari games, without making use of distributional RL, n-step returns or prioritized replay. To demonstrate the versatility of this idea, we also use it together with an Implicit Quantile Network (IQN). The resulting agent outperforms Rainbow on Atari, installing a new State of the Art with very little modifications to the original algorithm. To add to this empirical study, we provide strong theoretical insights on what happens under the hood -- implicit Kullback-Leibler regularization and increase of the action-gap.

arXiv commentNeurIPS 2020. Code: https://github.com/google-research/google-research/tree/master/munchausen_rl

Cited by in corpus (14)

• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Adversarially Guided Actor-Critic
• Self-Imitation Advantage Learning
• Globally Optimal Hierarchical Reinforcement Learning for Linearly-Solvable Markov Decision Processes
• Continuous Control with Action Quantization from Demonstrations
• Dual Behavior Regularized Reinforcement Learning
• Implicitly Regularized RL with Implicit Q-Values
• Logistic Q-Learning
• There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning
• Intentionally-underestimated Value Function at Terminal State for Temporal-difference Learning with Mis-designed Reward
• Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learning
• Geometric Value Iteration: Dynamic Error-Aware KL Regularization for Reinforcement Learning
• Regularize! Don't Mix: Multi-Agent Reinforcement Learning without Explicit Centralized Structures