A Regularized Approach to Sparse Optimal Policy in Reinforcement Learning

Abstract

We propose and study a general framework for regularized Markov decision processes (MDPs) where the goal is to find an optimal policy that maximizes the expected discounted total reward plus a policy regularization term. The extant entropy-regularized MDPs can be cast into our framework. Moreover, under our framework, many regularization terms can bring multi-modality and sparsity, which are potentially useful in reinforcement learning. In particular, we present sufficient and necessary conditions that induce a sparse optimal policy. We also conduct a full mathematical analysis of the proposed regularized MDPs, including the optimality condition, performance error, and sparseness control. We provide a generic method to devise regularization forms and propose off-policy actor critic algorithms in complex environment settings. We empirically analyze the numerical properties of optimal policies and compare the performance of different sparse regularization forms in discrete and continuous environments.

Cited by in corpus (6)

• Reinforcement Learning-based Visual Navigation with Information-Theoretic Regularization
• Relative Sparsity for Medical Decision Problems
• Divergence-Regularized Multi-Agent Actor-Critic
• Regularized Inverse Reinforcement Learning
• Regularized Policies are Reward Robust
• Reinforcement Learning With Sparse-Executing Actions via Sparsity Regularization