Lift-the-flap: what, where and when for context reasoning
arXiv:1902.00163
Abstract
Context reasoning is critical in a wide variety of applications where current inputs need to be interpreted in the light of previous experience and knowledge. Both spatial and temporal contextual information play a critical role in the domain of visual recognition. Here we investigate spatial constraints (what image features provide contextual information and where they are located), and temporal constraints (when different contextual cues matter) for visual recognition. The task is to reason about the scene context and infer what a target object hidden behind a flap is in a natural image. To tackle this problem, we first describe an online human psychophysics experiment recording active sampling via mouse clicks in lift-the-flap games and identify clicking patterns and features which are diagnostic for high contextual reasoning accuracy. As a proof of the usefulness of these clicking patterns and visual features, we extend a state-of-the-art recurrent model capable of attending to salient context regions, dynamically integrating useful information, making inferences, and predicting class label for the target object over multiple clicks. The proposed model achieves human-level contextual reasoning accuracy, shares human-like sampling behavior and learns interpretable features for contextual reasoning.
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Deep Residual Learning for Image Recognition
- Conditional Random Fields as Recurrent Neural Networks
- Recurrent Neural Network Regularization
- Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
- Multiple Object Recognition with Visual Attention
- Interaction Networks for Learning about Objects, Relations and Physics
- Fully Connected Deep Structured Networks
- Recurrent computations for visual pattern completion
- Finding any Waldo: zero-shot invariant and efficient visual search
- Deep Reinforcement Learning for Robotic Manipulation-The state of the art
- The More You Know: Using Knowledge Graphs for Image Classification