Fusion of Satellite Images and Weather Data with Transformer Networks for Downy Mildew Disease Detection

Abstract

Crop diseases significantly affect the quantity and quality of agricultural production. In a context where the goal of precision agriculture is to minimize or even avoid the use of pesticides, weather and remote sensing data with deep learning can play a pivotal role in detecting crop diseases, allowing localized treatment of crops. However, combining heterogeneous data such as weather and images remains a hot topic and challenging task. Recent developments in transformer architectures have shown the possibility of fusion of data from different domains, for instance text-image. The current trend is to custom only one transformer to create a multimodal fusion model. Conversely, we propose a new approach to realize data fusion using three transformers. In this paper, we first solved the missing satellite images problem, by interpolating them with a ConvLSTM model. Then, proposed a multimodal fusion architecture that jointly learns to process visual and weather information. The architecture is built from three main components, a Vision Transformer and two transformer-encoders, allowing to fuse both image and weather modalities. The results of the proposed method are promising achieving 97\% overall accuracy.

References in corpus (6)

• Remote Sensing Image Change Detection with Transformers
• ResViT: Residual vision transformers for multi-modal medical image synthesis
• Multimodal Fusion Transformer for Remote Sensing Image Classification
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• Attention Bottlenecks for Multimodal Fusion
• VddNet: Vine Disease Detection Network Based on Multispectral Images and Depth Map