Embodied Visual Recognition

Abstract

Passive visual systems typically fail to recognize objects in the amodal setting where they are heavily occluded. In contrast, humans and other embodied agents have the ability to move in the environment, and actively control the viewing angle to better understand object shapes and semantics. In this work, we introduce the task of Embodied Visual Recognition (EVR): An agent is instantiated in a 3D environment close to an occluded target object, and is free to move in the environment to perform object classification, amodal object localization, and amodal object segmentation. To address this, we develop a new model called Embodied Mask R-CNN, for agents to learn to move strategically to improve their visual recognition abilities. We conduct experiments using the House3D environment. Experimental results show that: 1) agents with embodiment (movement) achieve better visual recognition performance than passive ones; 2) in order to improve visual recognition abilities, agents can learn strategical moving paths that are different from shortest paths.

14 pages, 13 figures, technical report

References in corpus (9)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• YOLOv3: An Incremental Improvement
• Joint 2D-3D-Semantic Data for Indoor Scene Understanding
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Building Generalizable Agents with a Realistic and Rich 3D Environment
• MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
• HoME: a Household Multimodal Environment
• Semantic Amodal Segmentation

Cited by in corpus (7)

• VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects
• O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning
• Learning Object-conditioned Exploration using Distributed Soft Actor Critic
• Identifying Model Weakness with Adversarial Examiner
• Towards Embodied Scene Description
• EWareNet: Emotion Aware Human Intent Prediction and Adaptive Spatial Profile Fusion for Social Robot Navigation
• Spatiotemporal Attacks for Embodied Agents