Self-attention for raw optical Satellite Time Series Classification

Abstract

The amount of available Earth observation data has increased dramatically in the recent years. Efficiently making use of the entire body information is a current challenge in remote sensing and demands for light-weight problem-agnostic models that do not require region- or problem-specific expert knowledge. End-to-end trained deep learning models can make use of raw sensory data by learning feature extraction and classification in one step solely from data. Still, many methods proposed in remote sensing research require implicit feature extraction through data preprocessing or explicit design of features. In this work, we compare recent deep learning models on crop type classification on raw and preprocessed Sentinel 2 data. We concentrate on the common neural network architectures for time series, i.e., 1D-convolutions, recurrence, a shallow random forest baseline, and focus on the novel self-attention architecture. Our central findings are that data preprocessing still increased the overall classification performance for all models while the choice of model was less crucial. Self-attention and recurrent neural networks, by their architecture, outperformed convolutional neural networks on raw satellite time series. We explore this by a feature importance analysis based on gradient back-propagation that exploits the differentiable nature of deep learning models. Further, we qualitatively show how self-attention scores focus selectively on few classification-relevant observations.

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Scikit-learn: Machine Learning in Python
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• Deep learning for time series classification: a review
• Remote Sensing Image Scene Classification: Benchmark and State of the Art
• InceptionTime: Finding AlexNet for Time Series Classification
• Learning Spectral-Spatial-Temporal Features via a Recurrent Convolutional Neural Network for Change Detection in Multispectral Imagery
• Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data
• Tune: A Research Platform for Distributed Model Selection and Training
• Multi-Scale Convolutional Neural Networks for Time Series Classification
• Dense semantic labeling of sub-decimeter resolution images with convolutional neural networks
• Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders
• Fully Convolutional Networks for Dense Semantic Labelling of High-Resolution Aerial Imagery
• Land Cover Classification from Multi-temporal, Multi-spectral Remotely Sensed Imagery using Patch-Based Recurrent Neural Networks
• The UEA multivariate time series classification archive, 2018
• Classification of Time-Series Images Using Deep Convolutional Neural Networks

Cited by in corpus (24)

• Transformers in Time-series Analysis: A Tutorial
• Deep Learning for Satellite Image Time Series Analysis: A Review
• Gated Transformer Networks for Multivariate Time Series Classification
• Adaptive Fusion of Multi-view Remote Sensing data for Optimal Sub-field Crop Yield Prediction
• Opening the Black-Box: A Systematic Review on Explainable AI in Remote Sensing
• TimeMatch: Unsupervised Cross-Region Adaptation by Temporal Shift Estimation
• A Sentinel-2 multi-year, multi-country benchmark dataset for crop classification and segmentation with deep learning
• U-TILISE: A Sequence-to-sequence Model for Cloud Removal in Optical Satellite Time Series
• Explainability of Sub-Field Level Crop Yield Prediction using Remote Sensing
• A systematic review of the use of Deep Learning in Satellite Imagery for Agriculture
• Early Detection of Forest Calamities in Homogeneous Stands -- Deep Learning Applied to Bark-Beetle Outbreaks
• Lightweight Temporal Self-Attention for Classifying Satellite Image Time Series
• Harnessing Administrative Data Inventories to Create a Reliable Transnational Reference Database for Crop Type Monitoring
• XAI for Early Crop Classification
• Domain-Adversarial Training of Self-Attention Based Networks for Land Cover Classification using Multi-temporal Sentinel-2 Satellite Imagery
• Evaluating the Role of Training Data Origin for Country-Scale Cropland Mapping in Data-Scarce Regions: A Case Study of Nigeria
• AI-driven Orchestration at Scale: Estimating Service Metrics on National-Wide Testbeds
• From Time-series Generation, Model Selection to Transfer Learning: A Comparative Review of Pixel-wise Approaches for Large-scale Crop Mapping
• An Efficient Method for the Classification of Croplands in Scarce-Label Regions
• Benchmarking for Practice: Few-Shot Time-Series Crop-Type Classification on the EuroCropsML Dataset
• Channel-Based Attention for LCC Using Sentinel-2 Time Series
• In the Search for Optimal Multi-view Learning Models for Crop Classification with Global Remote Sensing Data
• Crop Rotation Modeling for Deep Learning-Based Parcel Classification from Satellite Time Series
• Bayesian Modelling of Multi-Year Crop Type Classification Using Deep Neural Networks and Hidden Markov Models