Rearrangement: A Challenge for Embodied AI

Abstract

We describe a framework for research and evaluation in Embodied AI. Our proposal is based on a canonical task: Rearrangement. A standard task can focus the development of new techniques and serve as a source of trained models that can be transferred to other settings. In the rearrangement task, the goal is to bring a given physical environment into a specified state. The goal state can be specified by object poses, by images, by a description in language, or by letting the agent experience the environment in the goal state. We characterize rearrangement scenarios along different axes and describe metrics for benchmarking rearrangement performance. To facilitate research and exploration, we present experimental testbeds of rearrangement scenarios in four different simulation environments. We anticipate that other datasets will be released and new simulation platforms will be built to support training of rearrangement agents and their deployment on physical systems.

arXiv commentAuthors are listed in alphabetical order

References in corpus (9)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Sequence to Sequence Learning with Neural Networks
• Universal Robotic Gripper based on the Jamming of Granular Material
• On Evaluation of Embodied Navigation Agents
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• AI2-THOR: An Interactive 3D Environment for Visual AI
• MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
• FutureMapping: The Computational Structure of Spatial AI Systems
• Synchronized Multi-Arm Rearrangement Guided by Mode Graphs with Capacity Constraints

Cited by in corpus (26)

• DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics
• iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks
• Challenges for Monocular 6D Object Pose Estimation in Robotics
• From Screens to Scenes: A Survey of Embodied AI in Healthcare
• N${}^2$M${}^2$: Learning Navigation for Arbitrary Mobile Manipulation Motions in Unseen and Dynamic Environments
• ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
• Habitat 2.0: Training Home Assistants to Rearrange their Habitat
• MQA: Answering the Question via Robotic Manipulation
• Recent Advances of Deep Robotic Affordance Learning: A Reinforcement Learning Perspective
• Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks
• The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AI
• Predicting Stable Configurations for Semantic Placement of Novel Objects
• CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents
• OCRTOC: A Cloud-Based Competition and Benchmark for Robotic Grasping and Manipulation
• Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion
• MANER: Multi-Agent Neural Rearrangement Planning of Objects in Cluttered Environments
• Efficient learning of goal-oriented push-grasping synergy in clutter
• LUMINOUS: Indoor Scene Generation for Embodied AI Challenges
• Shaping embodied agent behavior with activity-context priors from egocentric video
• My House, My Rules: Learning Tidying Preferences with Graph Neural Networks
• Discovering Generalizable Skills via Automated Generation of Diverse Tasks
• Scene Editing as Teleoperation: A Case Study in 6DoF Kit Assembly
• Evaluating model-based planning and planner amortization for continuous control
• AI-Driven Relocation Tracking in Dynamic Kitchen Environments
• Indoor Semantic Scene Understanding using Multi-modality Fusion
• Pushing it out of the Way: Interactive Visual Navigation