SAITS: Self-Attention-based Imputation for Time Series

Abstract

Missing data in time series is a pervasive problem that puts obstacles in the way of advanced analysis. A popular solution is imputation, where the fundamental challenge is to determine what values should be filled in. This paper proposes SAITS, a novel method based on the self-attention mechanism for missing value imputation in multivariate time series. Trained by a joint-optimization approach, SAITS learns missing values from a weighted combination of two diagonally-masked self-attention (DMSA) blocks. DMSA explicitly captures both the temporal dependencies and feature correlations between time steps, which improves imputation accuracy and training speed. Meanwhile, the weighted-combination design enables SAITS to dynamically assign weights to the learned representations from two DMSA blocks according to the attention map and the missingness information. Extensive experiments quantitatively and qualitatively demonstrate that SAITS outperforms the state-of-the-art methods on the time-series imputation task efficiently and reveal SAITS' potential to improve the learning performance of pattern recognition models on incomplete time-series data from the real world. The code is open source on GitHub at https://github.com/WenjieDu/SAITS.

This paper has been published in the journal Expert Systems with Applications. The DOI link is https://doi.org/10.1016/j.eswa.2023.119619/

References in corpus (1)

• Forecasting Loss of Signal in Optical Networks with Machine Learning

Cited by in corpus (19)

• ImputeFormer: Low Rankness-Induced Transformers for Generalizable Spatiotemporal Imputation
• MTSCI: A Conditional Diffusion Model for Multivariate Time Series Consistent Imputation
• Networked Time Series Imputation via Position-aware Graph Enhanced Variational Autoencoders
• Self-Supervised Learning of Time Series Representation via Diffusion Process and Imputation-Interpolation-Forecasting Mask
• Joint Estimation and Prediction of City-wide Delivery Demand: A Large Language Model Empowered Graph-based Learning Approach
• Boundary-enhanced time series data imputation with long-term dependency diffusion models
• RDIS: Random Drop Imputation with Self-Training for Incomplete Time Series Data
• SSD-TS: Exploring the Potential of Linear State Space Models for Diffusion Models in Time Series Imputation
• A novel feature selection framework for incomplete data
• Missing Data as Augmentation in the Earth Observation Domain: A Multi-View Learning Approach
• Iterative missing value imputation based on feature importance
• ADF & TransApp: A Transformer-Based Framework for Appliance Detection Using Smart Meter Consumption Series
• Multi-scale Restoration of Missing Data in Optical Time-series Images with Masked Spatial-Temporal Attention Network
• Filling in the Blanks: Applying Data Imputation in incomplete Water Metering Data
• Time Series Continuous Modeling for Imputation and Forecasting with Implicit Neural Representations
• Deep sub-ensembles meets quantile regression: uncertainty-aware imputation for time series
• STDiff: A State Transition Diffusion Framework for Time Series Imputation in Industrial Systems
• Measurements of Cosmic Proton Flux through Neutron Monitors Using Deep Networks and Imputation Techniques in the AMS-02 Era
• An Analysis of Temporal Dropout in Earth Observation Time Series for Regression Tasks