Reward Shaping via Meta-Learning

Abstract

Reward shaping is one of the most effective methods to tackle the crucial yet challenging problem of credit assignment in Reinforcement Learning (RL). However, designing shaping functions usually requires much expert knowledge and hand-engineering, and the difficulties are further exacerbated given multiple similar tasks to solve. In this paper, we consider reward shaping on a distribution of tasks, and propose a general meta-learning framework to automatically learn the efficient reward shaping on newly sampled tasks, assuming only shared state space but not necessarily action space. We first derive the theoretically optimal reward shaping in terms of credit assignment in model-free RL. We then propose a value-based meta-learning algorithm to extract an effective prior over the optimal reward shaping. The prior can be applied directly to new tasks, or provably adapted to the task-posterior while solving the task within few gradient updates. We demonstrate the effectiveness of our shaping through significantly improved learning efficiency and interpretable visualizations across various settings, including notably a successful transfer from DQN to DDPG.

first two authors contributed equally

References in corpus (6)

• Matching Networks for One Shot Learning
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Learning to reinforcement learn
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Hierarchical Macro Strategy Model for MOBA Game AI

Cited by in corpus (14)

• Meta-learning PINN loss functions
• Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Learning to Generalize from Sparse and Underspecified Rewards
• A Tutorial on Ultra-Reliable and Low-Latency Communications in 6G: Integrating Domain Knowledge into Deep Learning
• Self-Attentional Credit Assignment for Transfer in Reinforcement Learning
• Balance Between Efficient and Effective Learning: Dense2Sparse Reward Shaping for Robot Manipulation with Environment Uncertainty
• Predictive Coding for Boosting Deep Reinforcement Learning with Sparse Rewards
• Programming and Deployment of Autonomous Swarms using Multi-Agent Reinforcement Learning
• Embedding Contextual Information through Reward Shaping in Multi-Agent Learning: A Case Study from Google Football
• Connecting Context-specific Adaptation in Humans to Meta-learning
• HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem
• Learning to Shape Rewards using a Game of Two Partners
• Model-Augmented Q-learning