Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learning

Abstract

Goal-conditioned policies are used in order to break down complex reinforcement learning (RL) problems by using subgoals, which can be defined either in state space or in a latent feature space. This can increase the efficiency of learning by using a curriculum, and also enables simultaneous learning and generalization across goals. A crucial requirement of goal-conditioned policies is to be able to determine whether the goal has been achieved. Having a notion of distance to a goal is thus a crucial component of this approach. However, it is not straightforward to come up with an appropriate distance, and in some tasks, the goal space may not even be known a priori. In this work we learn a distance-to-goal estimate which is computed in terms of the number of actions that would need to be carried out in a self-supervised approach. Our method solves complex tasks without prior domain knowledge in the online setting in three different scenarios in the context of goal-conditioned policies a) the goal space is the same as the state space b) the goal space is given but an appropriate distance is unknown and c) the state space is accessible, but only a subset of the state space represents desired goals, and this subset is known a priori. We also propose a goal-generation mechanism as a secondary contribution.

arXiv commentPreprint; Under Review (updated)

References in corpus (9)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Visual Reinforcement Learning with Imagined Goals
• Episodic Curiosity through Reachability
• Reverse Curriculum Generation for Reinforcement Learning
• Hindsight policy gradients
• Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning

Cited by in corpus (7)

• A survey on intrinsic motivation in reinforcement learning
• Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• Adversarial Intrinsic Motivation for Reinforcement Learning
• Outcome-Driven Reinforcement Learning via Variational Inference
• Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning
• C-Learning: Horizon-Aware Cumulative Accessibility Estimation