Learning Latent Plans from Play

Abstract

Acquiring a diverse repertoire of general-purpose skills remains an open challenge for robotics. In this work, we propose self-supervising control on top of human teleoperated play data as a way to scale up skill learning. Play has two properties that make it attractive compared to conventional task demonstrations. Play is cheap, as it can be collected in large quantities quickly without task segmenting, labeling, or resetting to an initial state. Play is naturally rich, covering ~4x more interaction space than task demonstrations for the same amount of collection time. To learn control from play, we introduce Play-LMP, a self-supervised method that learns to organize play behaviors in a latent space, then reuse them at test time to achieve specific goals. Combining self-supervised control with a diverse play dataset shifts the focus of skill learning from a narrow and discrete set of tasks to the full continuum of behaviors available in an environment. We find that this combination generalizes well empirically---after self-supervising on unlabeled play, our method substantially outperforms individual expert-trained policies on 18 difficult user-specified visual manipulation tasks in a simulated robotic tabletop environment. We additionally find that play-supervised models, unlike their expert-trained counterparts, are more robust to perturbations and exhibit retrying-till-success behaviors. Finally, we find that our agent organizes its latent plan space around functional tasks, despite never being trained with task labels. Videos, code and data are available at learning-from-play.github.io

Published at CoRL 2019 (3rd Conference on Robot Learning, Osaka, Japan)

References in corpus (20)

• Continuous control with deep reinforcement learning
• Neural Discrete Representation Learning
• Soft Actor-Critic Algorithms and Applications
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Hindsight Experience Replay
• Visual Reinforcement Learning with Imagined Goals
• One-Shot Visual Imitation Learning via Meta-Learning
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Unsupervised Learning of Visual Representations using Videos
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Learning to Poke by Poking: Experiential Learning of Intuitive Physics
• Self-Supervised Visual Planning with Temporal Skip Connections
• Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation
• Hindsight policy gradients
• Deep Visual Foresight for Planning Robot Motion
• The Intentional Unintentional Agent: Learning to Solve Many Continuous Control Tasks Simultaneously

Cited by in corpus (18)

• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
• Learning to Reach Goals via Iterated Supervised Learning
• RoboNet: Large-Scale Multi-Robot Learning
• Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• Time Reversal as Self-Supervision
• On Multi-Agent Learning in Team Sports Games
• Evaluating Agents without Rewards
• Learning Multi-Stage Tasks with One Demonstration via Self-Replay
• Situated GAIL: Multitask imitation using task-conditioned adversarial inverse reinforcement learning
• The Differentiable Cross-Entropy Method
• Learning Visually Guided Latent Actions for Assistive Teleoperation
• SAFARI: Safe and Active Robot Imitation Learning with Imagination
• Robotic Imitation of Human Assembly Skills Using Hybrid Trajectory and Force Learning
• Machine Learning for Robotic Manipulation
• Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning
• ToolNet: Using Commonsense Generalization for Predicting Tool Use for Robot Plan Synthesis
• Distilling a Hierarchical Policy for Planning and Control via Representation and Reinforcement Learning