Conditioning of Reinforcement Learning Agents and its Policy Regularization Application

Abstract

The outcome of Jacobian singular values regularization was studied for supervised learning problems. It also was shown that Jacobian conditioning regularization can help to avoid the ``mode-collapse'' problem in Generative Adversarial Networks. In this paper, we try to answer the following question: Can information about policy conditioning help to shape a more stable and general policy of reinforcement learning agents? To answer this question, we conduct a study of Jacobian conditioning behavior during policy optimization. To the best of our knowledge, this is the first work that research condition number in reinforcement learning agents. We propose a conditioning regularization algorithm and test its performance on the range of continuous control tasks. Finally, we compare algorithms on the CoinRun environment with separated train end test levels to analyze how conditioning regularization contributes to agents' generalization.

References in corpus (8)

• Asynchronous Methods for Deep Reinforcement Learning
• Improved Techniques for Training GANs
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• A Study on Overfitting in Deep Reinforcement Learning
• Generalization and Regularization in DQN
• Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck
• Information asymmetry in KL-regularized RL