Visual Semantic Planning using Deep Successor Representations

Abstract

A crucial capability of real-world intelligent agents is their ability to plan a sequence of actions to achieve their goals in the visual world. In this work, we address the problem of visual semantic planning: the task of predicting a sequence of actions from visual observations that transform a dynamic environment from an initial state to a goal state. Doing so entails knowledge about objects and their affordances, as well as actions and their preconditions and effects. We propose learning these through interacting with a visual and dynamic environment. Our proposed solution involves bootstrapping reinforcement learning with imitation learning. To ensure cross task generalization, we develop a deep predictive model based on successor representations. Our experimental results show near optimal results across a wide range of tasks in the challenging THOR environment.

ICCV 2017 camera ready

Cited by in corpus (9)

• Motion-Appearance Co-Memory Networks for Video Question Answering
• Guided Feature Transformation (GFT): A Neural Language Grounding Module for Embodied Agents
• Cross-Domain Facial Expression Recognition: A Unified Evaluation Benchmark and Adversarial Graph Learning
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
• Probing Emergent Semantics in Predictive Agents via Question Answering
• Learning to guide task and motion planning using score-space representation
• NavigationNet: A Large-scale Interactive Indoor Navigation Dataset
• Sidekick Policy Learning for Active Visual Exploration
• Deep Reinforcement Learning with Label Embedding Reward for Supervised Image Hashing