HoME: a Household Multimodal Environment

Abstract

We introduce HoME: a Household Multimodal Environment for artificial agents to learn from vision, audio, semantics, physics, and interaction with objects and other agents, all within a realistic context. HoME integrates over 45,000 diverse 3D house layouts based on the SUNCG dataset, a scale which may facilitate learning, generalization, and transfer. HoME is an open-source, OpenAI Gym-compatible platform extensible to tasks in reinforcement learning, language grounding, sound-based navigation, robotics, multi-agent learning, and more. We hope HoME better enables artificial agents to learn as humans do: in an interactive, multimodal, and richly contextualized setting.

Presented at NIPS 2017's Visually-Grounded Interaction and Language Workshop

References in corpus (3)

• Matterport3D: Learning from RGB-D Data in Indoor Environments
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• Grounded Language Learning in a Simulated 3D World

Cited by in corpus (33)

• On Evaluation of Embodied Navigation Agents
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
• ViZDoom Competitions: Playing Doom from Pixels
• CHALET: Cornell House Agent Learning Environment
• The StreetLearn Environment and Dataset
• Deep Learning Based 3D Segmentation: A Survey
• A Comprehensive Review of Modern Object Segmentation Approaches
• Natural Environment Benchmarks for Reinforcement Learning
• Embodied Visual Recognition
• Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research
• Collaborative Visual Navigation
• VirtualHome: Simulating Household Activities via Programs
• The Regretful Agent: Heuristic-Aided Navigation through Progress Estimation
• Embodied Question Answering in Photorealistic Environments with Point Cloud Perception
• REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments
• Learning Affordance Landscapes for Interaction Exploration in 3D Environments
• MLM: A Benchmark Dataset for Multitask Learning with Multiple Languages and Modalities
• Interactive Language Learning by Question Answering
• Why Build an Assistant in Minecraft?
• CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents
• Multi-Target Embodied Question Answering
• Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
• SoundSpaces: Audio-Visual Navigation in 3D Environments
• SAILenv: Learning in Virtual Visual Environments Made Simple
• Mastering emergent language: learning to guide in simulated navigation
• SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Driving
• Explore the Potential Performance of Vision-and-Language Navigation Model: a Snapshot Ensemble Method
• ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments
• Continual Reinforcement Learning with Diversity Exploration and Adversarial Self-Correction
• Incremental Scene Synthesis
• Language coverage and generalization in RNN-based continuous sentence embeddings for interacting agents