Action Redundancy in Reinforcement Learning

Abstract

Maximum Entropy (MaxEnt) reinforcement learning is a powerful learning paradigm which seeks to maximize return under entropy regularization. However, action entropy does not necessarily coincide with state entropy, e.g., when multiple actions produce the same transition. Instead, we propose to maximize the transition entropy, i.e., the entropy of next states. We show that transition entropy can be described by two terms; namely, model-dependent transition entropy and action redundancy. Particularly, we explore the latter in both deterministic and stochastic settings and develop tractable approximation methods in a near model-free setup. We construct algorithms to minimize action redundancy and demonstrate their effectiveness on a synthetic environment with multiple redundant actions as well as contemporary benchmarks in Atari and Mujoco. Our results suggest that action redundancy is a fundamental problem in reinforcement learning.

arXiv commentEqual Contribution. In Proceedings UAI 2021

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• A Survey on Multi-Task Learning
• Reinforcement Learning with Deep Energy-Based Policies
• Hindsight Experience Replay
• FeUdal Networks for Hierarchical Reinforcement Learning
• Variational Intrinsic Control
• Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning
• Provably Efficient Maximum Entropy Exploration
• Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning
• Geometric Entropic Exploration
• Autonomous exploration for navigating in non-stationary CMPs
• Geometric Losses for Distributional Learning
• Fast Reinforcement Learning with Large Action Sets using Error-Correcting Output Codes for MDP Factorization