AI2-THOR: An Interactive 3D Environment for Visual AI
arXiv:1712.05474
Abstract
We introduce The House Of inteRactions (THOR), a framework for visual AI research, available at http://ai2thor.allenai.org. AI2-THOR consists of near photo-realistic 3D indoor scenes, where AI agents can navigate in the scenes and interact with objects to perform tasks. AI2-THOR enables research in many different domains including but not limited to deep reinforcement learning, imitation learning, learning by interaction, planning, visual question answering, unsupervised representation learning, object detection and segmentation, and learning models of cognition. The goal of AI2-THOR is to facilitate building visually intelligent models and push the research forward in this domain.
References in corpus (9)
- Matterport3D: Learning from RGB-D Data in Indoor Environments
- Virtual Worlds as Proxy for Multi-Object Tracking Analysis
- Building Generalizable Agents with a Realistic and Rich 3D Environment
- MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
- SceneNet RGB-D: 5M Photorealistic Images of Synthetic Indoor Trajectories with Ground Truth
- TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games
- HoME: a Household Multimodal Environment
- Learning Physical Intuition of Block Towers by Example
- DeepMind Lab
Cited by in corpus (141)
- On the Opportunities and Risks of Foundation Models
- Unity: A General Platform for Intelligent Agents
- On Evaluation of Embodied Navigation Agents
- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
- robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
- Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
- ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
- Vision-and-Dialog Navigation
- Rearrangement: A Challenge for Embodied AI
- ViZDoom Competitions: Playing Doom from Pixels
- Spatial Action Maps for Mobile Manipulation
- Unity Perception: Generate Synthetic Data for Computer Vision
- A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation
- iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks
- Imitating Latent Policies from Observation
- Learning to Navigate in Cities Without a Map
- The StreetLearn Environment and Dataset
- EvalAI: Towards Better Evaluation Systems for AI Agents
- AllenAct: A Framework for Embodied AI Research
- iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes
- Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
- PlaTe: Visually-Grounded Planning with Transformers in Procedural Tasks
- Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
- BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments
- VTNet: Visual Transformer Network for Object Goal Navigation
- Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI
- VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning
- Simulating Content Consistent Vehicle Datasets with Attribute Descent
- Blindfold Baselines for Embodied QA
- Learning hierarchical relationships for object-goal navigation
- ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
- Natural Environment Benchmarks for Reinforcement Learning
- ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
- A Survey of Embodied AI: From Simulators to Research Tasks
- Habitat 2.0: Training Home Assistants to Rearrange their Habitat
- The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation
- LanguageRefer: Spatial-Language Model for 3D Visual Grounding
- Embodied Visual Recognition
- Improving Target-driven Visual Navigation with Attention on 3D Spatial Relationships
- Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Data
- Learning Object Relation Graph and Tentative Policy for Visual Navigation
- Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training
- Collaborative Visual Navigation
- CraftAssist: A Framework for Dialogue-enabled Interactive Agents
- CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning
- Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of others
- A Mask-RCNN Baseline for Probabilistic Object Detection
- Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
- VirtualHome: Simulating Household Activities via Programs
- Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration
- From Seeing to Moving: A Survey on Learning for Visual Indoor Navigation (VIN)
- FILM: Following Instructions in Language with Modular Methods
- Embodied Question Answering in Photorealistic Environments with Point Cloud Perception
- The Regretful Agent: Heuristic-Aided Navigation through Progress Estimation
- 3DB: A Framework for Debugging Computer Vision Models
- REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments
- OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets
- Deep Learning for Embodied Vision Navigation: A Survey
- What Should I Do Now? Marrying Reinforcement Learning and Symbolic Planning
- Learning Affordance Landscapes for Interaction Exploration in 3D Environments
- Cross-Domain Facial Expression Recognition: A Unified Evaluation Benchmark and Adversarial Graph Learning
- Learning Generalizable Visual Representations via Interactive Gameplay
- Evaluating the Impact of Semantic Segmentation and Pose Estimation on Dense Semantic SLAM
- Synthesized Policies for Transfer and Adaptation across Tasks and Environments
- Multi-View Learning for Vision-and-Language Navigation
- The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AI
- Why Build an Assistant in Minecraft?
- VideoNavQA: Bridging the Gap between Visual and Embodied Question Answering
- Look, Listen, and Act: Towards Audio-Visual Embodied Navigation
- CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents
- BenchBot: Evaluating Robotics Research in Photorealistic 3D Simulation and on Real Robots
- NViSII: A Scriptable Tool for Photorealistic Image Generation
- Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
- Fever Basketball: A Complex, Flexible, and Asynchronized Sports Game Environment for Multi-agent Reinforcement Learning
- Multi-Target Embodied Question Answering
- SoundSpaces: Audio-Visual Navigation in 3D Environments
- Sim-Real Joint Reinforcement Transfer for 3D Indoor Navigation
- Are We There Yet? Learning to Localize in Embodied Instruction Following
- Bayesian Relational Memory for Semantic Visual Navigation
- No RL, No Simulation: Learning to Navigate without Navigating
- An A* Curriculum Approach to Reinforcement Learning for RGBD Indoor Robot Navigation
- Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring
- Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigation
- Unsupervised Domain Adaptation for Visual Navigation
- SAILenv: Learning in Virtual Visual Environments Made Simple
- Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion
- Environment Predictive Coding for Embodied Agents
- Discovery of Options via Meta-Learned Subgoals
- Navigation Agents for the Visually Impaired: A Sidewalk Simulator and Experiments
- Target Driven Visual Navigation with Hybrid Asynchronous Universal Successor Representations
- Learning About Objects by Learning to Interact with Them
- Learning to Map for Active Semantic Goal Navigation
- GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving
- The Robotic Vision Scene Understanding Challenge
- SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environments
- LUMINOUS: Indoor Scene Generation for Embodied AI Challenges
- Zero-Shot Compositional Policy Learning via Language Grounding
- Learning to simulate complex scenes
- Vision-Dialog Navigation by Exploring Cross-modal Memory
- Large Batch Simulation for Deep Reinforcement Learning
- Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies
- Bridging the Imitation Gap by Adaptive Insubordination
- Shaping embodied agent behavior with activity-context priors from egocentric video
- ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments
- Adaptive Procedural Task Generation for Hard-Exploration Problems
- Explore the Potential Performance of Vision-and-Language Navigation Model: a Snapshot Ensemble Method
- Learning to Generate Synthetic 3D Training Data through Hybrid Gradient
- Cross-View Policy Learning for Street Navigation
- Walking with MIND: Mental Imagery eNhanceD Embodied QA
- Learning Adaptive Language Interfaces through Decomposition
- Optimal Assistance for Object-Rearrangement Tasks in Augmented Reality
- MaAST: Map Attention with Semantic Transformersfor Efficient Visual Navigation
- SIGVerse: A cloud-based VR platform for research on social and embodied human-robot interaction
- Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
- Landmark Policy Optimization for Object Navigation Task
- Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
- NavigationNet: A Large-scale Interactive Indoor Navigation Dataset
- Bridging Scene Understanding and Task Execution with Flexible Simulation Environments
- SeanNet: Semantic Understanding Network for Localization Under Object Dynamics
- Learning Autonomous Exploration and Mapping with Semantic Vision
- Discovering Generalizable Skills via Automated Generation of Diverse Tasks
- Multi-Agent Embodied Visual Semantic Navigation with Scene Prior Knowledge
- Dialogue Object Search
- Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation
- Multimodal Aggregation Approach for Memory Vision-Voice Indoor Navigation with Meta-Learning
- VRGym: A Virtual Testbed for Physical and Interactive AI
- Language coverage and generalization in RNN-based continuous sentence embeddings for interacting agents
- Spatiotemporal Attacks for Embodied Agents
- Pushing it out of the Way: Interactive Visual Navigation
- MINERVAS: Massive INterior EnviRonments VirtuAl Synthesis
- Messing Up 3D Virtual Environments: Transferable Adversarial 3D Objects
- Contextual Scene Augmentation and Synthesis via GSACNet
- Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments
- Hierarchical and Partially Observable Goal-driven Policy Learning with Goals Relational Graph
- VSGM -- Enhance robot task understanding ability through visual semantic graph
- Leveraging Semantics for Incremental Learning in Multi-Relational Embeddings
- Synthetic Data Are as Good as the Real for Association Knowledge Learning in Multi-object Tracking
- Active Object Perceiver: Recognition-guided Policy Learning for Object Searching on Mobile Robots
- Enhanced Scene Specificity with Sparse Dynamic Value Estimation
- Robot in a China Shop: Using Reinforcement Learning for Location-Specific Navigation Behaviour
- Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in a First-person Simulated 3D Environment