VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation

Abstract

Robotic fabric manipulation has applications in home robotics, textiles, senior care and surgery. Existing fabric manipulation techniques, however, are designed for specific tasks, making it difficult to generalize across different but related tasks. We extend the Visual Foresight framework to learn fabric dynamics that can be efficiently reused to accomplish different fabric manipulation tasks with a single goal-conditioned policy. We introduce VisuoSpatial Foresight (VSF), which builds on prior work by learning visual dynamics on domain randomized RGB images and depth maps simultaneously and completely in simulation. We experimentally evaluate VSF on multi-step fabric smoothing and folding tasks against 5 baseline methods in simulation and on the da Vinci Research Kit (dVRK) surgical robot without any demonstrations at train or test time. Furthermore, we find that leveraging depth significantly improves performance. RGBD data yields an 80% improvement in fabric folding success rate over pure RGB data. Code, data, videos, and supplementary material are available at https://sites.google.com/view/fabric-vsf/.

Robotics: Science and Systems (RSS) 2020

References in corpus (8)

• Continuous control with deep reinforcement learning
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Safe Controller Optimization for Quadrotors with Gaussian Processes
• Learning Dexterous In-Hand Manipulation
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Cautious NMPC with Gaussian Process Dynamics for Autonomous Miniature Race Cars
• On-Policy Robot Imitation Learning from a Converging Supervisor
• Time Reversal as Self-Supervision

Cited by in corpus (9)

• Accelerating Surgical Robotics Research: A Review of 10 Years With the da Vinci Research Kit
• In-air Knotting of Rope using Dual-Arm Robot based on Deep Learning
• Learning Arbitrary-Goal Fabric Folding with One Hour of Real Robot Experience
• Learning Dense Visual Correspondences in Simulation to Smooth and Fold Real Fabrics
• VisuoSpatial Foresight for Physical Sequential Fabric Manipulation
• Untangling Dense Knots by Learning Task-Relevant Keypoints
• Machine Learning for Robotic Manipulation
• Cloth Manipulation Planning on Basis of Mesh Representations with Incomplete Domain Knowledge and Voxel-to-Mesh Estimation
• Visual Perspective Taking for Opponent Behavior Modeling