Long-term Forecasting using Higher Order Tensor RNNs

Abstract

We present Higher-Order Tensor RNN (HOT-RNN), a novel family of neural sequence architectures for multivariate forecasting in environments with nonlinear dynamics. Long-term forecasting in such systems is highly challenging, since there exist long-term temporal dependencies, higher-order correlations and sensitivity to error propagation. Our proposed recurrent architecture addresses these issues by learning the nonlinear dynamics directly using higher-order moments and higher-order state transition functions. Furthermore, we decompose the higher-order structure using the tensor-train decomposition to reduce the number of parameters while preserving the model performance. We theoretically establish the approximation guarantees and the variance bound for HOT-RNN for general sequence inputs. We also demonstrate 5% ~ 12% improvements for long-term prediction over general RNN and LSTM architectures on a range of simulated environments with nonlinear dynamics, as well on real-world time series data.

24 pages including appendix, updated JMLR version

References in corpus (18)

• Deep Learning in Neural Networks: An Overview
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Unsupervised Learning of Video Representations using LSTMs
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Tensorizing Neural Networks
• Hierarchical Multiscale Recurrent Neural Networks
• Spectral tensor-train decomposition
• Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition
• Tensor-Train Recurrent Neural Networks for Video Classification
• Higher Order Recurrent Neural Networks
• Learning from Multiway Data: Simple and Efficient Tensor Regression
• Learning to Reason with Third-Order Tensor Products
• Nonparametric Estimation of Multi-View Latent Variable Models
• Expressive power of recurrent neural networks
• Practical Learning of Predictive State Representations
• Exponential Families for Conditional Random Fields

Cited by in corpus (33)

• Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
• Deep Learning Methods for Reynolds-Averaged Navier-Stokes Simulations of Airfoil Flows
• Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• Bayesian Temporal Factorization for Multidimensional Time Series Prediction
• Convolutional Tensor-Train LSTM for Spatio-temporal Learning
• Tensorized Embedding Layers for Efficient Model Compression
• Tensor Train decomposition on TensorFlow (T3F)
• Connecting Weighted Automata and Recurrent Neural Networks through Spectral Learning
• Explainable Tensorized Neural Ordinary Differential Equations forArbitrary-step Time Series Prediction
• Foundations of Sequence-to-Sequence Modeling for Time Series
• Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition
• Bayesian Tensorized Neural Networks with Automatic Rank Selection
• Identifying nonlinear dynamical systems with multiple time scales and long-range dependencies
• A Multi-view Multi-task Learning Framework for Multi-variate Time Series Forecasting
• Expressive power of recurrent neural networks
• Generating Realistic Stock Market Order Streams
• Towards Extremely Compact RNNs for Video Recognition with Fully Decomposed Hierarchical Tucker Structure
• MQTransformer: Multi-Horizon Forecasts with Context Dependent and Feedback-Aware Attention
• Tensorized Transformer for Dynamical Systems Modeling
• Long-term series forecasting with Query Selector -- efficient model of sparse attention
• A Deep Learning Framework for COVID Outbreak Prediction
• Quantum Tensor Networks, Stochastic Processes, and Weighted Automata
• H-OWAN: Multi-distorted Image Restoration with Tensor 1x1 Convolution
• Structural Forecasting for Tropical Cyclone Intensity Prediction: Providing Insight with Deep Learning
• Tensor-Train Networks for Learning Predictive Modeling of Multidimensional Data
• On the Memory Mechanism of Tensor-Power Recurrent Models
• Bridging the Gap Between Training and Inference for Spatio-Temporal Forecasting
• Block Hankel Tensor ARIMA for Multiple Short Time Series Forecasting
• Physics-based polynomial neural networks for one-shot learning of dynamical systems from one or a few samples
• HECT: High-Dimensional Ensemble Consistency Testing for Climate Models
• Block-term Tensor Neural Networks
• A Sampling-Based Method for Tensor Ring Decomposition