Multimodal Reward Shaping for Efficient Exploration in Reinforcement Learning

Abstract

Maintaining the long-term exploration capability of the agent remains one of the critical challenges in deep reinforcement learning. A representative solution is to leverage reward shaping to provide intrinsic rewards for the agent to encourage exploration. However, most existing methods suffer from vanishing intrinsic rewards, which cannot provide sustainable exploration incentives. Moreover, they rely heavily on complex models and additional memory to record learning procedures, resulting in high computational complexity and low robustness. To tackle this problem, entropy-based methods are proposed to evaluate the global exploration performance, encouraging the agent to visit the state space more equitably. However, the sample complexity of estimating the state visitation entropy is prohibitive when handling environments with high-dimensional observations. In this paper, we introduce a novel metric entitled Jain's fairness index (JFI) to replace the entropy regularizer, which solves the exploration problem from a brand new perspective. In sharp contrast to the entropy regularizer, JFI is more computable and robust and can be easily applied generalized into arbitrary tasks. Furthermore, we leverage a variational auto-encoder (VAE) model to capture the life-long novelty of states, which is combined with the global JFI score to form multimodal intrinsic rewards. Finally, extensive simulation results demonstrate that our multimodal reward shaping (MMRS) method can achieve higher performance than other benchmark schemes.

arXiv comment17 pages, 5 figures

References in corpus (12)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Large-Scale Study of Curiosity-Driven Learning
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Exploration by Random Network Distillation
• Go-Explore: a New Approach for Hard-Exploration Problems
• Episodic Curiosity through Reachability
• Efficient Exploration via State Marginal Matching
• A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
• State Entropy Maximization with Random Encoders for Efficient Exploration
• Intrinsic Reward Driven Imitation Learning via Generative Model
• Entropy Regularization with Discounted Future State Distribution in Policy Gradient Methods