Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks

Abstract

Rearranging and manipulating deformable objects such as cables, fabrics, and bags is a long-standing challenge in robotic manipulation. The complex dynamics and high-dimensional configuration spaces of deformables, compared to rigid objects, make manipulation difficult not only for multi-step planning, but even for goal specification. Goals cannot be as easily specified as rigid object poses, and may involve complex relative spatial relations such as "place the item inside the bag". In this work, we develop a suite of simulated benchmarks with 1D, 2D, and 3D deformable structures, including tasks that involve image-based goal-conditioning and multi-step deformable manipulation. We propose embedding goal-conditioning into Transporter Networks, a recently proposed model architecture for learning robotic manipulation that rearranges deep features to infer displacements that can represent pick and place actions. In simulation and in physical experiments, we demonstrate that goal-conditioned Transporter Networks enable agents to manipulate deformable structures into flexibly specified configurations without test-time visual anchors for target locations. We also significantly extend prior results using Transporter Networks for manipulating deformable objects by testing on tasks with 2D and 3D deformables. Supplementary material is available at https://berkeleyautomation.github.io/bags/.

arXiv commentSee https://berkeleyautomation.github.io/bags/ for project website and code; v3 is ICRA 2021 version and v4 adds physical experiments and improves simulation results

References in corpus (9)

• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation
• Rearrangement: A Challenge for Embodied AI
• Learning Predictive Representations for Deformable Objects Using Contrastive Estimation
• Spatial Action Maps for Mobile Manipulation
• SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation
• Learning Arbitrary-Goal Fabric Folding with One Hour of Real Robot Experience
• TossingBot: Learning to Throw Arbitrary Objects with Residual Physics