A Geometric Perspective on Self-Supervised Policy Adaptation

Abstract

One of the most challenging aspects of real-world reinforcement learning (RL) is the multitude of unpredictable and ever-changing distractions that could divert an agent from what was tasked to do in its training environment. While an agent could learn from reward signals to ignore them, the complexity of the real-world can make rewards hard to acquire, or, at best, extremely sparse. A recent class of self-supervised methods have shown promise that reward-free adaptation under challenging distractions is possible. However, previous work focused on a short one-episode adaptation setting. In this paper, we consider a long-term adaptation setup that is more akin to the specifics of the real-world and propose a geometric perspective on self-supervised adaptation. We empirically describe the processes that take place in the embedding space during this adaptation process, reveal some of its undesirable effects on performance and show how they can be eliminated. Moreover, we theoretically study how actor-based and actor-free agents can further generalise to the target environment by manipulating the geometry of the manifolds described by the actor and critic functions.

arXiv commentContains 17 pages, 18 figures

References in corpus (9)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• DeepMind Control Suite
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Learning Invariant Representations for Reinforcement Learning without Reconstruction
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• Self-Supervised Policy Adaptation during Deployment
• Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning
• The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels