Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning
arXiv:2110.12997
Abstract
Unsupervised reinforcement learning aims to acquire skills without prior goal representations, where an agent automatically explores an open-ended environment to represent goals and learn the goal-conditioned policy. However, this procedure is often time-consuming, limiting the rollout in some potentially expensive target environments. The intuitive approach of training in another interaction-rich environment disrupts the reproducibility of trained skills in the target environment due to the dynamics shifts and thus inhibits direct transferring. Assuming free access to a source environment, we propose an unsupervised domain adaptation method to identify and acquire skills across dynamics. Particularly, we introduce a KL regularized objective to encourage emergence of skills, rewarding the agent for both discovering skills and aligning its behaviors respecting dynamics shifts. This suggests that both dynamics (source and target) shape the reward to facilitate the learning of adaptive skills. We also conduct empirical experiments to demonstrate that our method can effectively learn skills that can be smoothly deployed in target.
arXiv commentAccepted by NeurIPS 2021
References in corpus (15)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
- DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
- Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
- End-to-End Robotic Reinforcement Learning without Reward Engineering
- Learning to Share and Hide Intentions using Information Regularization
- Unsupervised Curricula for Visual Meta-Reinforcement Learning
- Action and Perception as Divergence Minimization
- State Alignment-based Imitation Learning
- Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
- Domain Adaptive Imitation Learning
- Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
- Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learning
- Disentangled Skill Embeddings for Reinforcement Learning
- Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch