Self-Supervised Policy Adaptation during Deployment

Abstract

In most real world scenarios, a policy trained by reinforcement learning in one environment needs to be deployed in another, potentially quite different environment. However, generalization across different environments is known to be hard. A natural solution would be to keep training after deployment in the new environment, but this cannot be done if the new environment offers no reward signal. Our work explores the use of self-supervision to allow the policy to continue training after deployment without using any rewards. While previous methods explicitly anticipate changes in the new environment, we assume no prior knowledge of those changes yet still obtain significant improvements. Empirical evaluations are performed on diverse simulation environments from DeepMind Control suite and ViZDoom, as well as real robotic manipulation tasks in continuously changing environments, taking observations from an uncalibrated camera. Our method improves generalization in 31 out of 36 environments across various tasks and outperforms domain randomization on a majority of environments.

Website: https://nicklashansen.github.io/PAD/ Code: https://github.com/nicklashansen/policy-adaptation-during-deployment ICLR 2021

References in corpus (27)

• A Simple Framework for Contrastive Learning of Visual Representations
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic Algorithms and Applications
• Unsupervised Domain Adaptation with Residual Transfer Networks
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• DeepMind Control Suite
• Recurrent World Models Facilitate Policy Evolution
• Generalisation in humans and deep neural networks
• Do ImageNet Classifiers Generalize to ImageNet?
• Robust Adversarial Reinforcement Learning
• Visual Reinforcement Learning with Imagined Goals
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Reinforcement Learning with Augmented Data
• Unsupervised Domain Adaptation through Self-Supervision
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?
• Assessing Generalization in Deep Reinforcement Learning
• Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations
• ViZDoom Competitions: Playing Doom from Pixels
• Learning Predictive Representations for Deformable Objects Using Contrastive Estimation
• Planning to Explore via Self-Supervised World Models
• Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning
• InGAN: Capturing and Remapping the "DNA" of a Natural Image
• Domain Generalization via Conditional Invariant Representation
• Single Episode Policy Transfer in Reinforcement Learning

Cited by in corpus (14)

• A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
• Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation
• Generalization in Reinforcement Learning by Soft Data Augmentation
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• Learning Task-relevant Representations for Generalization via Characteristic Functions of Reward Sequence Distributions
• The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels
• Transformers for One-Shot Visual Imitation
• Exploratory State Representation Learning
• Domain Adversarial Reinforcement Learning
• Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research
• Unsupervised Visual Attention and Invariance for Reinforcement Learning
• A Geometric Perspective on Self-Supervised Policy Adaptation