Adversarial Intrinsic Motivation for Reinforcement Learning
arXiv:2105.13345
Abstract
Learning with an objective to minimize the mismatch with a reference distribution has been shown to be useful for generative modeling and imitation learning. In this paper, we investigate whether one such objective, the Wasserstein-1 distance between a policy's state visitation distribution and a target distribution, can be utilized effectively for reinforcement learning (RL) tasks. Specifically, this paper focuses on goal-conditioned reinforcement learning where the idealized (unachievable) target distribution has full measure at the goal. This paper introduces a quasimetric specific to Markov Decision Processes (MDPs) and uses this quasimetric to estimate the above Wasserstein-1 distance. It further shows that the policy that minimizes this Wasserstein-1 distance is the policy that reaches the goal in as few steps as possible. Our approach, termed Adversarial Intrinsic Motivation (AIM), estimates this Wasserstein-1 distance through its dual objective and uses it to compute a supplemental reward function. Our experiments show that this reward function changes smoothly with respect to transitions in the MDP and directs the agent's exploration to find the goal efficiently. Additionally, we combine AIM with Hindsight Experience Replay (HER) and show that the resulting algorithm accelerates learning significantly on several simulated robotics tasks when compared to other rewards that encourage exploration or accelerate learning.
References in corpus (21)
- Continuous control with deep reinforcement learning
- Reinforcement Learning with Deep Energy-Based Policies
- Hindsight Experience Replay
- Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
- Synthesizing Programs for Images using Reinforced Adversarial Learning
- RUDDER: Return Decomposition for Delayed Rewards
- From optimal transport to generative modeling: the VEGAN cookbook
- Provably Efficient Maximum Entropy Exploration
- Generative Adversarial Imitation from Observation
- Efficient Exploration via State Marginal Matching
- Primal Wasserstein Imitation Learning
- Wasserstein Adversarial Imitation Learning
- Automatic Curriculum Learning through Value Disagreement
- On Learning Intrinsic Rewards for Policy Gradient Methods
- Wasserstein GANs Work Because They Fail (to Approximate the Wasserstein Distance)
- SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments
- Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learning
- C-Learning: Learning to Achieve Goals via Recursive Classification
- What Can Learned Intrinsic Rewards Capture?
- Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
- An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning