Time Reversal as Self-Supervision

Abstract

A longstanding challenge in robot learning for manipulation tasks has been the ability to generalize to varying initial conditions, diverse objects, and changing objectives. Learning based approaches have shown promise in producing robust policies, but require heavy supervision to efficiently learn precise control, especially from visual inputs. We propose a novel self-supervision technique that uses time-reversal to learn goals and provide a high level plan to reach them. In particular, we introduce the time-reversal model (TRM), a self-supervised model which explores outward from a set of goal states and learns to predict these trajectories in reverse. This provides a high level plan towards goals, allowing us to learn complex manipulation tasks with no demonstrations or exploration at test time. We test our method on the domain of assembly, specifically the mating of tetris-style block pairs. Using our method operating atop visual model predictive control, we are able to assemble tetris blocks on a physical robot using only uncalibrated RGB camera input, and generalize to unseen block pairs. sites.google.com/view/time-reversal

arXiv comment7 pages, 10 figures

References in corpus (18)

• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Hindsight Experience Replay
• Distributed Distributional Deterministic Policy Gradients
• Visual Reinforcement Learning with Imagined Goals
• Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Reverse Curriculum Generation for Reinforcement Learning
• Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task
• Self-Supervised Visual Planning with Temporal Skip Connections
• Sim-to-Real Reinforcement Learning for Deformable Object Manipulation
• Stochastic Variational Video Prediction
• SE3-Pose-Nets: Structured Deep Dynamics Models for Visuomotor Planning and Control
• Recall Traces: Backtracking Models for Efficient Reinforcement Learning
• Learning Latent Plans from Play
• Deep Visual Foresight for Planning Robot Motion
• Neural Task Programming: Learning to Generalize Across Hierarchical Tasks
• Deep Predictive Policy Training using Reinforcement Learning

Cited by in corpus (8)

• VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation
• RoboNet: Large-Scale Multi-Robot Learning
• Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation
• Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly
• Robot Program Parameter Inference via Differentiable Shadow Program Inversion
• VisuoSpatial Foresight for Physical Sequential Fabric Manipulation
• Learning the Arrow of Time
• There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning