ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation

Abstract

We introduce ThreeDWorld (TDW), a platform for interactive multi-modal physical simulation. TDW enables simulation of high-fidelity sensory data and physical interactions between mobile agents and objects in rich 3D environments. Unique properties include: real-time near-photo-realistic image rendering; a library of objects and environments, and routines for their customization; generative procedures for efficiently building classes of new environments; high-fidelity audio rendering; realistic physical interactions for a variety of material types, including cloths, liquid, and deformable objects; customizable agents that embody AI agents; and support for human interactions with VR devices. TDW's API enables multiple agents to interact within a simulation and returns a range of sensor and physics data representing the state of the world. We present initial experiments enabled by TDW in emerging research directions in computer vision, machine learning, and cognitive science, including multi-modal physical scene understanding, physical dynamics predictions, multi-agent interactions, models that learn like a child, and attention studies in humans and neural networks.

arXiv commentOral Presentation at NeurIPS 21 Datasets and Benchmarks Track. Project page: http://www.threedworld.org

References in corpus (9)

• Fine-Grained Visual Classification of Aircraft
• Interaction Networks for Learning about Objects, Relations and Physics
• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Building Generalizable Agents with a Realistic and Rich 3D Environment
• A Compositional Object-Based Approach to Learning Physical Dynamics
• Flexible Neural Representation for Physics Prediction
• Learning Visual Predictive Models of Physics for Playing Billiards
• IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning

Cited by in corpus (26)

• On the Opportunities and Risks of Foundation Models
• Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments
• iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks
• AllenAct: A Framework for Embodied AI Research
• iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes
• Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
• ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
• A Survey of Embodied AI: From Simulators to Research Tasks
• Habitat 2.0: Training Home Assistants to Rearrange their Habitat
• PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics
• The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers
• Physion: Evaluating Physical Prediction from Vision in Humans and Machines
• Foley Music: Learning to Generate Music from Videos
• The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AI
• CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents
• LUMINOUS: Indoor Scene Generation for Embodied AI Challenges
• Shaping embodied agent behavior with activity-context priors from egocentric video
• Learning for Deformable Linear Object Insertion Leveraging Flexibility Estimation from Visual Cues
• CUDA-Optimized real-time rendering of a Foveated Visual System
• Large Batch Simulation for Deep Reinforcement Learning
• Caregiver Talk Shapes Toddler Vision: A Computational Study of Dyadic Play
• Learning a metacognition for object perception
• AGENT: A Benchmark for Core Psychological Reasoning
• Beyond Flatland: Pre-training with a Strong 3D Inductive Bias
• Messing Up 3D Virtual Environments: Transferable Adversarial 3D Objects
• Can machines learn to see without visual databases?