PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation

Abstract

The ability for robots to comprehend and execute manipulation tasks based on natural language instructions is a long-term goal in robotics. The dominant approaches for language-guided manipulation use 2D image representations, which face difficulties in combining multi-view cameras and inferring precise 3D positions and relationships. To address these limitations, we propose a 3D point cloud based policy called PolarNet for language-guided manipulation. It leverages carefully designed point cloud inputs, efficient point cloud encoders, and multimodal transformers to learn 3D point cloud representations and integrate them with language instructions for action prediction. PolarNet is shown to be effective and data efficient in a variety of experiments conducted on the RLBench benchmark. It outperforms state-of-the-art 2D and 3D approaches in both single-task and multi-task learning. It also achieves promising results on a real robot.

arXiv commentAccepted to CoRL 2023. Project website: https://www.di.ens.fr/willow/research/polarnet/

References in corpus (27)

• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Open3D: A Modern Library for 3D Data Processing
• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
• PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies
• PaLM-E: An Embodied Multimodal Language Model
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection
• CLIPort: What and Where Pathways for Robotic Manipulation
• BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
• VIMA: General Robot Manipulation with Multimodal Prompts
• Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation
• Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model
• Multimodal Masked Autoencoders Learn Transferable Representations
• Language-Conditioned Imitation Learning for Robot Manipulation Tasks
• Interactive Language: Talking to Robots in Real Time
• Auto-Lambda: Disentangling Dynamic Task Relationships
• VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation
• ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding
• Language Conditioned Spatial Relation Reasoning for 3D Object Grounding
• Learning Obstacle Representations for Neural Motion Planning
• DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation
• Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Clouds
• ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds
• Instruction-Following Agents with Multimodal Transformer
• Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction
• SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learning
• FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects