TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting

Abstract

Transformers have gained popularity in time series forecasting for their ability to capture long-sequence interactions. However, their high memory and computing requirements pose a critical bottleneck for long-term forecasting. To address this, we propose TSMixer, a lightweight neural architecture exclusively composed of multi-layer perceptron (MLP) modules for multivariate forecasting and representation learning on patched time series. Inspired by MLP-Mixer's success in computer vision, we adapt it for time series, addressing challenges and introducing validated components for enhanced accuracy. This includes a novel design paradigm of attaching online reconciliation heads to the MLP-Mixer backbone, for explicitly modeling the time-series properties such as hierarchy and channel-correlations. We also propose a novel Hybrid channel modeling and infusion of a simple gating approach to effectively handle noisy channel interactions and generalization across diverse datasets. By incorporating these lightweight components, we significantly enhance the learning capability of simple MLP structures, outperforming complex Transformer models with minimal computing usage. Moreover, TSMixer's modular design enables compatibility with both supervised and masked self-supervised learning methods, making it a promising building block for time-series Foundation Models. TSMixer outperforms state-of-the-art MLP and Transformer models in forecasting by a considerable margin of 8-60%. It also outperforms the latest strong benchmarks of Patch-Transformer models (by 1-2%) with a significant reduction in memory and runtime (2-3X). The source code of our model is officially released as PatchTSMixer in the HuggingFace. Model: https://huggingface.co/docs/transformers/main/en/model_doc/patchtsmixer Examples: https://github.com/ibm/tsfm/#notebooks-links

Accepted in the Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 23), Research Track. Delayed release in arXiv to comply with the conference policies on the double-blind review process. This paper has been submitted to the KDD peer-review process on Feb 02, 2023

References in corpus (5)

• MLP-Mixer: An all-MLP Architecture for Vision
• Less Is More: Fast Multivariate Time Series Forecasting with Light Sampling-oriented MLP Structures
• Unsupervised Time-Series Representation Learning with Iterative Bilinear Temporal-Spectral Fusion
• S${}^2$-MLP: Spatial-Shift MLP Architecture for Vision
• ResMLP: Feedforward networks for image classification with data-efficient training

Cited by in corpus (14)

• Deep Learning for Time Series Forecasting: A Survey
• Time Series Modeling for Heart Rate Prediction: From ARIMA to Transformers
• Supervised Contrastive Learning based Dual-Mixer Model for Remaining Useful Life Prediction
• MARLP: Time-series Forecasting Control for Agricultural Managed Aquifer Recharge
• Scalable Transformer for High Dimensional Multivariate Time Series Forecasting
• Proactive Model Adaptation Against Concept Drift for Online Time Series Forecasting
• CrossLinear: Plug-and-Play Cross-Correlation Embedding for Time Series Forecasting with Exogenous Variables
• Navigating Weight Prediction with Diet Diary
• A Backbone for Long-Horizon Robot Task Understanding
• Explainable Time Series Prediction of Tyre Energy in Formula One Race Strategy
• A Comprehensive Benchmark for Electrocardiogram Time-Series
• Explainable and Interpretable Forecasts on Non-Smooth Multivariate Time Series for Responsible Gameplay
• Enhancing Exchange Rate Forecasting with Explainable Deep Learning Models
• Timing is Important: Risk-aware Fund Allocation based on Time-Series Forecasting