Aerial Scene Understanding in The Wild: Multi-Scene Recognition via Prototype-based Memory Networks

Abstract

Aerial scene recognition is a fundamental visual task and has attracted an increasing research interest in the last few years. Most of current researches mainly deploy efforts to categorize an aerial image into one scene-level label, while in real-world scenarios, there often exist multiple scenes in a single image. Therefore, in this paper, we propose to take a step forward to a more practical and challenging task, namely multi-scene recognition in single images. Moreover, we note that manually yielding annotations for such a task is extraordinarily time- and labor-consuming. To address this, we propose a prototype-based memory network to recognize multiple scenes in a single image by leveraging massive well-annotated single-scene images. The proposed network consists of three key components: 1) a prototype learning module, 2) a prototype-inhabiting external memory, and 3) a multi-head attention-based memory retrieval module. To be more specific, we first learn the prototype representation of each aerial scene from single-scene aerial image datasets and store it in an external memory. Afterwards, a multi-head attention-based memory retrieval module is devised to retrieve scene prototypes relevant to query multi-scene images for final predictions. Notably, only a limited number of annotated multi-scene images are needed in the training phase. To facilitate the progress of aerial scene recognition, we produce a new multi-scene aerial image (MAI) dataset. Experimental results on variant dataset configurations demonstrate the effectiveness of our network. Our dataset and codes are publicly available.

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Prototypical Networks for Few-shot Learning
• End-To-End Memory Networks
• BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understanding
• Land cover mapping at very high resolution with rotation equivariant CNNs: towards small yet accurate models
• Recurrently Exploring Class-wise Attention in A Hybrid Convolutional and Bidirectional LSTM Network for Multi-label Aerial Image Classification
• IM2HEIGHT: Height Estimation from Single Monocular Imagery via Fully Residual Convolutional-Deconvolutional Network
• Multi-temporal and multi-source remote sensing image classification by nonlinear relative normalization
• RiFCN: Recurrent Network in Fully Convolutional Network for Semantic Segmentation of High Resolution Remote Sensing Images
• A Novel Multi-Attention Driven System For Multi-Label Remote Sensing Image Classification
• Remote Sensing Image Scene Classification with Deep Neural Networks in JPEG 2000 Compressed Domain
• Cross-Task Transfer for Geotagged Audiovisual Aerial Scene Recognition
• DAWN: Dual Augmented Memory Network for Unsupervised Video Object Tracking