CLIPort: What and Where Pathways for Robotic Manipulation

Abstract

How can we imbue robots with the ability to manipulate objects precisely but also to reason about them in terms of abstract concepts? Recent works in manipulation have shown that end-to-end networks can learn dexterous skills that require precise spatial reasoning, but these methods often fail to generalize to new goals or quickly learn transferable concepts across tasks. In parallel, there has been great progress in learning generalizable semantic representations for vision and language by training on large-scale internet data, however these representations lack the spatial understanding necessary for fine-grained manipulation. To this end, we propose a framework that combines the best of both worlds: a two-stream architecture with semantic and spatial pathways for vision-based manipulation. Specifically, we present CLIPort, a language-conditioned imitation-learning agent that combines the broad semantic understanding (what) of CLIP [1] with the spatial precision (where) of Transporter [2]. Our end-to-end framework is capable of solving a variety of language-specified tabletop tasks from packing unseen objects to folding cloths, all without any explicit representations of object poses, instance segmentations, memory, symbolic states, or syntactic structures. Experiments in simulated and real-world settings show that our approach is data efficient in few-shot settings and generalizes effectively to seen and unseen semantic concepts. We even learn one multi-task policy for 10 simulated and 9 real-world tasks that is better or comparable to single-task policies.

CoRL 2021. Project Website: https://cliport.github.io/

References in corpus (8)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Learning Transferable Visual Models From Natural Language Supervision
• Solving Rubik's Cube with a Robot Hand
• Group Equivariant Convolutional Networks
• Audiovisual SlowFast Networks for Video Recognition
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• A Joint Network for Grasp Detection Conditioned on Natural Language Commands

Cited by in corpus (13)

• Large AI Models in Health Informatics: Applications, Challenges, and the Future
• Text2Motion: From Natural Language Instructions to Feasible Plans
• DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics
• CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory
• "No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy
• Deep Reinforcement Learning Based on Local GNN for Goal-conditioned Deformable Object Rearranging
• A Simple Long-Tailed Recognition Baseline via Vision-Language Model
• LEMMA: Learning Language-Conditioned Multi-Robot Manipulation
• TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Models
• Graph-Transporter: A Graph-based Learning Method for Goal-Conditioned Deformable Object Rearranging Task
• PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation
• Visually Grounded Continual Language Learning with Selective Specialization
• Tool as Embodiment for Recursive Manipulation