An Information-Theoretic Optimality Principle for Deep Reinforcement Learning

Abstract

We methodologically address the problem of Q-value overestimation in deep reinforcement learning to handle high-dimensional state spaces efficiently. By adapting concepts from information theory, we introduce an intrinsic penalty signal encouraging reduced Q-value estimates. The resultant algorithm encompasses a wide range of learning outcomes containing deep Q-networks as a special case. Different learning outcomes can be demonstrated by tuning a Lagrange multiplier accordingly. We furthermore propose a novel scheduling scheme for this Lagrange multiplier to ensure efficient and robust learning. In experiments on Atari, our algorithm outperforms other algorithms (e.g. deep and double deep Q-networks) in terms of both game-play performance and sample complexity. These results remain valid under the recently proposed dueling architecture.

arXiv commentPresented at the NIPS Deep Reinforcement Learning Workshop, Montreal, Canada, 2018

References in corpus (10)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• The information bottleneck method
• Dueling Network Architectures for Deep Reinforcement Learning
• On the Convergence of Adam and Beyond
• Optimal control as a graphical model inference problem
• Thermodynamics as a theory of decision-making with information processing costs
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem
• Bounded Rational Decision-Making in Feedforward Neural Networks

Cited by in corpus (8)

• Bounded rational decision-making from elementary computations that reduce uncertainty
• A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment
• Bounded Rational Decision-Making with Adaptive Neural Network Priors
• A Tutorial on Sparse Gaussian Processes and Variational Inference
• ${\alpha }^{\alpha }$-Rank: Practically Scaling $\alpha$-Rank through Stochastic Optimisation
• Mutual-Information Regularization in Markov Decision Processes and Actor-Critic Learning
• Hierarchical Expert Networks for Meta-Learning
• Multi-Task Generative Adversarial Nets with Shared Memory for Cross-Domain Coordination Control