Semantically-Aware Attentive Neural Embeddings for Image-based Visual Localization
arXiv:1812.03402
Abstract
We present an approach that combines appearance and semantic information for 2D image-based localization (2D-VL) across large perceptual changes and time lags. Compared to appearance features, the semantic layout of a scene is generally more invariant to appearance variations. We use this intuition and propose a novel end-to-end deep attention-based framework that utilizes multimodal cues to generate robust embeddings for 2D-VL. The proposed attention module predicts a shared channel attention and modality-specific spatial attentions to guide the embeddings to focus on more reliable image regions. We evaluate our model against state-of-the-art (SOTA) methods on three challenging localization datasets. We report an average (absolute) improvement of over current SOTA for 2D-VL. Furthermore, we present an extensive study demonstrating the contribution of each component of our model, showing -- and improvement from adding semantic information and our proposed attention module. We finally show the predicted attention maps to offer useful insights into our model.
Appearing in BMVC 2019
References in corpus (10)
- Adam: A Method for Stochastic Optimization
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- Deep Audio-Visual Speech Recognition
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Convolutional Neural Network-based Place Recognition
- On the Performance of ConvNet Features for Place Recognition
- Attention-based Pyramid Aggregation Network for Visual Place Recognition
- Training a Convolutional Neural Network for Appearance-Invariant Place Recognition
- Understanding Visual Ads by Aligning Symbols and Objects using Co-Attention