On Learning Intrinsic Rewards for Policy Gradient Methods

Abstract

In many sequential decision making tasks, it is challenging to design reward functions that help an RL agent efficiently learn behavior that is considered good by the agent designer. A number of different formulations of the reward-design problem, or close variants thereof, have been proposed in the literature. In this paper we build on the Optimal Rewards Framework of Singh et.al. that defines the optimal intrinsic reward function as one that when used by an RL agent achieves behavior that optimizes the task-specifying or extrinsic reward function. Previous work in this framework has shown how good intrinsic reward functions can be learned for lookahead search based planning agents. Whether it is possible to learn intrinsic reward functions for learning agents remains an open problem. In this paper we derive a novel algorithm for learning intrinsic rewards for policy-gradient based learning agents. We compare the performance of an augmented agent that uses our algorithm to provide additive intrinsic rewards to an A2C-based policy learner (for Atari games) and a PPO-based policy learner (for Mujoco domains) with a baseline agent that uses the same policy learners but with only extrinsic rewards. Our results show improved performance on most but not all of the domains.

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Benchmarking Deep Reinforcement Learning for Continuous Control
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models

Cited by in corpus (15)

• Meta-learning PINN loss functions
• Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Generative Adversarial Self-Imitation Learning
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning
• Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning
• Learning to Incentivize Other Learning Agents
• Deception in Social Learning: A Multi-Agent Reinforcement Learning Perspective
• Adversarial Intrinsic Motivation for Reinforcement Learning
• Reward Learning for Efficient Reinforcement Learning in Extractive Document Summarisation
• MDP Playground: An Analysis and Debug Testbed for Reinforcement Learning
• Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learning
• Learning to Ask Medical Questions using Reinforcement Learning
• Wasserstein Distance Maximizing Intrinsic Control