C-Learning: Learning to Achieve Goals via Recursive Classification

Abstract

We study the problem of predicting and controlling the future state distribution of an autonomous agent. This problem, which can be viewed as a reframing of goal-conditioned reinforcement learning (RL), is centered around learning a conditional probability density function over future states. Instead of directly estimating this density function, we indirectly estimate this density function by training a classifier to predict whether an observation comes from the future. Via Bayes' rule, predictions from our classifier can be transformed into predictions over future states. Importantly, an off-policy variant of our algorithm allows us to predict the future state distribution of a new policy, without collecting new experience. This variant allows us to optimize functionals of a policy's future state distribution, such as the density of reaching a particular goal state. While conceptually similar to Q-learning, our work lays a principled foundation for goal-conditioned RL as density estimation, providing justification for goal-conditioned methods used in prior work. This foundation makes hypotheses about Q-learning, including the optimal goal-sampling ratio, which we confirm experimentally. Moreover, our proposed method is competitive with prior goal-conditioned RL methods.

Accepted at ICLR 2021. Project website with videos (https://ben-eysenbach.github.io/c_learning/) and code (https://github.com/google-research/google-research/tree/master/c_learning) are online

References in corpus (10)

• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• Variational Inference using Implicit Distributions
• Planning with Goal-Conditioned Policies
• Generative Adversarial Nets from a Density Ratio Estimation Perspective
• Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning
• Diagnosing Bottlenecks in Deep Q-learning Algorithms
• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
• Goal-Aware Prediction: Learning to Model What Matters
• Reinforcement Learning without Ground-Truth State
• Policy Continuation with Hindsight Inverse Dynamics