RSVQA: Visual Question Answering for Remote Sensing Data

Abstract

This paper introduces the task of visual question answering for remote sensing data (RSVQA). Remote sensing images contain a wealth of information which can be useful for a wide range of tasks including land cover classification, object counting or detection. However, most of the available methodologies are task-specific, thus inhibiting generic and easy access to the information contained in remote sensing data. As a consequence, accurate remote sensing product generation still requires expert knowledge. With RSVQA, we propose a system to extract information from remote sensing data that is accessible to every user: we use questions formulated in natural language and use them to interact with the images. With the system, images can be queried to obtain high level information specific to the image content or relational dependencies between objects visible in the images. Using an automatic method introduced in this article, we built two datasets (using low and high resolution data) of image/question/answer triplets. The information required to build the questions and answers is queried from OpenStreetMap (OSM). The datasets can be used to train (when using supervised methods) and evaluate models to solve the RSVQA task. We report the results obtained by applying a model based on Convolutional Neural Networks (CNNs) for the visual part and on a Recurrent Neural Network (RNN) for the natural language part to this task. The model is trained on the two datasets, yielding promising results in both cases.

arXiv comment12 pages, Published in IEEE Transactions on Geoscience and Remote Sensing. Added one experiment and authors' biographies

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Deep learning in remote sensing: a review
• Deep Learning for Hyperspectral Image Classification: An Overview
• DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images
• Skip-Thought Vectors
• Recent Advances in Domain Adaptation for the Classification of Remote Sensing Data
• Dense semantic labeling of sub-decimeter resolution images with convolutional neural networks
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Learning to Count Objects in Natural Images for Visual Question Answering
• R${}^3$-Net: A Deep Network for Multi-oriented Vehicle Detection in Aerial Images and Videos
• Correcting rural building annotations in OpenStreetMap using convolutional neural networks

Cited by in corpus (20)

• RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data
• Language-aware Domain Generalization Network for Cross-Scene Hyperspectral Image Classification
• RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing
• From Easy to Hard: Learning Language-guided Curriculum for Visual Question Answering on Remote Sensing Data
• Towards a Collective Agenda on AI for Earth Science Data Analysis
• AI Security for Geoscience and Remote Sensing: Challenges and Future Trends
• Change Detection Meets Visual Question Answering
• Txt2Img-MHN: Remote Sensing Image Generation from Text Using Modern Hopfield Networks
• Remote Sensing SpatioTemporal Vision-Language Models: A Comprehensive Survey
• Earth System Data Cubes: Avenues for advancing Earth system research
• Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectives
• Recent Advances and Trends in Multimodal Deep Learning: A Review
• How Good is Google Bard's Visual Understanding? An Empirical Study on Open Challenges
• RSAdapter: Adapting Multimodal Models for Remote Sensing Visual Question Answering
• How to find a good image-text embedding for remote sensing visual question answering?
• Bounding Boxes Are All We Need: Street View Image Classification via Context Encoding of Detected Buildings
• Geographic Question Answering: Challenges, Uniqueness, Classification, and Future Directions
• RS-Agent: Automating Remote Sensing Tasks through Intelligent Agent
• ZeShot-VQA: Zero-Shot Visual Question Answering Framework with Answer Mapping for Natural Disaster Damage Assessment
• How Can Multimodal Remote Sensing Datasets Transform Classification via SpatialNet-ViT?