Self-Supervised Exploration via Disagreement

Abstract

Efficient exploration is a long-standing problem in sensorimotor learning. Major advances have been demonstrated in noise-free, non-stochastic domains such as video games and simulation. However, most of these formulations either get stuck in environments with stochastic dynamics or are too inefficient to be scalable to real robotics setups. In this paper, we propose a formulation for exploration inspired by the work in active learning literature. Specifically, we train an ensemble of dynamics models and incentivize the agent to explore such that the disagreement of those ensembles is maximized. This allows the agent to learn skills by exploring in a self-supervised manner without any external reward. Notably, we further leverage the disagreement objective to optimize the agent's policy in a differentiable manner, without using reinforcement learning, which results in a sample-efficient exploration. We demonstrate the efficacy of this formulation across a variety of benchmark environments including stochastic-Atari, Mujoco and Unity. Finally, we implement our differentiable exploration on a real robot which learns to interact with objects completely from scratch. Project videos and code are at https://pathak22.github.io/exploration-by-disagreement/

Accepted at ICML 2019. Website at https://pathak22.github.io/exploration-by-disagreement/

References in corpus (12)

• Continuous control with deep reinforcement learning
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Noisy Networks for Exploration
• Unity: A General Platform for Intelligent Agents
• Bayesian Active Learning for Classification and Preference Learning
• Parameter Space Noise for Exploration
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
• Diversity is All You Need: Learning Skills without a Reward Function
• Model-Based Active Exploration
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning

Cited by in corpus (14)

• Automatic Curriculum Learning through Value Disagreement
• Attention-based Curiosity-driven Exploration in Deep Reinforcement Learning
• BeBold: Exploration Beyond the Boundary of Explored Regions
• On the model-based stochastic value gradient for continuous reinforcement learning
• RoboNet: Large-Scale Multi-Robot Learning
• OffWorld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research
• Multimodal Safety-Critical Scenarios Generation for Decision-Making Algorithms Evaluation
• Exploratory State Representation Learning
• Exploring Exploration: Comparing Children with RL Agents in Unified Environments
• Towards a Hypothesis on Visual Transformation based Self-Supervision
• Co-Imitation Learning without Expert Demonstration
• Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning
• Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitation
• Self-Supervised Contextual Bandits in Computer Vision