Weakly-Supervised Reinforcement Learning for Controllable Behavior
arXiv:2004.02860
Abstract
Reinforcement learning (RL) is a powerful framework for learning to take actions to solve tasks. However, in many settings, an agent must winnow down the inconceivably large space of all possible tasks to the single task that it is currently being asked to solve. Can we instead constrain the space of tasks to those that are semantically meaningful? In this work, we introduce a framework for using weak supervision to automatically disentangle this semantically meaningful subspace of tasks from the enormous space of nonsensical "chaff" tasks. We show that this learned subspace enables efficient exploration and provides a representation that captures distance between states. On a variety of challenging, vision-based continuous control problems, our approach leads to substantial performance gains, particularly as the complexity of the environment grows.
arXiv commentPublished in NeurIPS 2020
References in corpus (36)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
- Dota 2 with Large Scale Deep Reinforcement Learning
- Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
- Disentangling by Factorising
- Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
- Model-Based Reinforcement Learning for Atari
- Hindsight Experience Replay
- Visual Reinforcement Learning with Imagined Goals
- Towards a Definition of Disentangled Representations
- Temporal Difference Models: Model-Free Deep RL for Model-Based Control
- Automatic Goal Generation for Reinforcement Learning Agents
- SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
- Are Disentangled Representations Helpful for Abstract Visual Reasoning?
- Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
- Provably Efficient Maximum Entropy Exploration
- Loss is its own Reward: Self-Supervision for Reinforcement Learning
- Efficient Exploration via State Marginal Matching
- A Laplacian Framework for Option Discovery in Reinforcement Learning
- Weakly Supervised Disentanglement with Guarantees
- DeepMDP: Learning Continuous Latent Space Models for Representation Learning
- Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
- Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
- Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
- Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition
- Semi-parametric Topological Memory for Navigation
- Independently Controllable Factors
- InfoBot: Transfer and Exploration via the Information Bottleneck
- RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation
- Independently Controllable Features
- Few-Shot Goal Inference for Visuomotor Learning and Planning
- Self-supervised Learning of Image Embedding for Continuous Control
- RoboNet: Large-Scale Multi-Robot Learning
- Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learning
- Demystifying Inter-Class Disentanglement