Weakly Supervised Disentangled Representation for Goal-conditioned Reinforcement Learning
arXiv:2202.13624 · doi:10.1109/LRA.2022.3141148
Abstract
Goal-conditioned reinforcement learning is a crucial yet challenging algorithm which enables agents to achieve multiple user-specified goals when learning a set of skills in a dynamic environment. However, it typically requires millions of the environmental interactions explored by agents, which is sample-inefficient. In the paper, we propose a skill learning framework DR-GRL that aims to improve the sample efficiency and policy generalization by combining the Disentangled Representation learning and Goal-conditioned visual Reinforcement Learning. In a weakly supervised manner, we propose a Spatial Transform AutoEncoder (STAE) to learn an interpretable and controllable representation in which different parts correspond to different object attributes (shape, color, position). Due to the high controllability of the representations, STAE can simply recombine and recode the representations to generate unseen goals for agents to practice themselves. The manifold structure of the learned representation maintains consistency with the physical position, which is beneficial for reward calculation. We empirically demonstrate that DR-GRL significantly outperforms the previous methods in sample efficiency and policy generalization. In addition, DR-GRL is also easy to expand to the real robot.
arXiv comment8 pages; Accepted by RAL with ICRA 2022
References in corpus (11)
- Auto-Encoding Variational Bayes
- mixup: Beyond Empirical Risk Minimization
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Neural Discrete Representation Learning
- Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
- Visual Reinforcement Learning with Imagined Goals
- Unsupervised Learning of Object Keypoints for Perception and Control
- Rigid-Soft Interactive Learning for Robust Grasping
- Sill-Net: Feature Augmentation with Separated Illumination Representation
- Dual Swap Disentangling
- ROLL: Visual Self-Supervised Reinforcement Learning with Object Reasoning