From Easy to Hard: Learning Language-guided Curriculum for Visual Question Answering on Remote Sensing Data

Abstract

Visual question answering (VQA) for remote sensing scene has great potential in intelligent human-computer interaction system. Although VQA in computer vision has been widely researched, VQA for remote sensing data (RSVQA) is still in its infancy. There are two characteristics that need to be specially considered for the RSVQA task. 1) No object annotations are available in RSVQA datasets, which makes it difficult for models to exploit informative region representation; 2) There are questions with clearly different difficulty levels for each image in the RSVQA task. Directly training a model with questions in a random order may confuse the model and limit the performance. To address these two problems, in this paper, a multi-level visual feature learning method is proposed to jointly extract language-guided holistic and regional image features. Besides, a self-paced curriculum learning (SPCL)-based VQA model is developed to train networks with samples in an easy-to-hard way. To be more specific, a language-guided SPCL method with a soft weighting strategy is explored in this work. The proposed model is evaluated on three public datasets, and extensive experimental results show that the proposed RSVQA framework can achieve promising performance.

References in corpus (14)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Deep learning in remote sensing: a review
• Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmark
• AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification
• A Survey on Object Detection in Optical Remote Sensing Images
• Learning Spectral-Spatial-Temporal Features via a Recurrent Convolutional Neural Network for Change Detection in Multispectral Imagery
• GETNET: A General End-to-end Two-dimensional CNN Framework for Hyperspectral Image Change Detection
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Land Use Classification in Remote Sensing Images by Convolutional Neural Networks
• RSVQA: Visual Question Answering for Remote Sensing Data
• ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
• Learning to Count Objects in Natural Images for Visual Question Answering
• Deep Learning applied to NLP
• Object-Centric Diagnosis of Visual Reasoning

Cited by in corpus (6)

• RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data
• Change Detection Meets Visual Question Answering
• Better, Not Just More: Data-Centric Machine Learning for Earth Observation
• RSAdapter: Adapting Multimodal Models for Remote Sensing Visual Question Answering
• How Can Multimodal Remote Sensing Datasets Transform Classification via SpatialNet-ViT?
• ZeShot-VQA: Zero-Shot Visual Question Answering Framework with Answer Mapping for Natural Disaster Damage Assessment