Towards Embodied Scene Description
arXiv:2004.14638
Abstract
Embodiment is an important characteristic for all intelligent agents (creatures and robots), while existing scene description tasks mainly focus on analyzing images passively and the semantic understanding of the scenario is separated from the interaction between the agent and the environment. In this work, we propose the Embodied Scene Description, which exploits the embodiment ability of the agent to find an optimal viewpoint in its environment for scene description tasks. A learning framework with the paradigms of imitation learning and reinforcement learning is established to teach the intelligent agent to generate corresponding sensorimotor activities. The proposed framework is tested on both the AI2Thor dataset and a real world robotic platform demonstrating the effectiveness and extendability of the developed method.
Accepted in Robotics: Science and Systems 2020
References in corpus (8)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Distributed Representations of Words and Phrases and their Compositionality
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
- Learning to Walk via Deep Reinforcement Learning
- Emergence of Exploratory Look-Around Behaviors through Active Observation Completion
- Embodied Visual Recognition
- OIL: Observational Imitation Learning