What Can Learned Intrinsic Rewards Capture?

Abstract

The objective of a reinforcement learning agent is to behave so as to maximise the sum of a suitable scalar function of state: the reward. These rewards are typically given and immutable. In this paper, we instead consider the proposition that the reward function itself can be a good locus of learned knowledge. To investigate this, we propose a scalable meta-gradient framework for learning useful intrinsic reward functions across multiple lifetimes of experience. Through several proof-of-concept experiments, we show that it is feasible to learn and capture knowledge about long-term exploration and exploitation into a reward function. Furthermore, we show that unlike policy transfer methods that capture "how" the agent should behave, the learned reward functions can generalise to other kinds of agents and to changes in the dynamics of the environment by capturing "what" the agent should strive to do.

ICML 2020. The first two authors contributed equally

References in corpus (11)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Learning to reinforcement learn
• Count-Based Exploration with Neural Density Models
• Meta-Gradient Reinforcement Learning
• Meta-Learning Update Rules for Unsupervised Representation Learning
• Learning to Understand Goal Specifications by Modelling Reward
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• Discovery of Useful Questions as Auxiliary Tasks
• Adapting Behaviour via Intrinsic Reward: A Survey and Empirical Study
• Deep Learning for Reward Design to Improve Monte Carlo Tree Search in ATARI Games

Cited by in corpus (3)

• Discovery of Options via Meta-Learned Subgoals
• Adversarial Intrinsic Motivation for Reinforcement Learning
• Wasserstein Distance Maximizing Intrinsic Control