Residual Attention Net for Superior Cross-Domain Time Sequence Modeling
arXiv:2001.04077
Abstract
We present a novel architecture, residual attention net (RAN), which merges a sequence architecture, universal transformer, and a computer vision architecture, residual net, with a high-way architecture for cross-domain sequence modeling. The architecture aims at addressing the long dependency issue often faced by recurrent-neural-net-based structures. This paper serves as a proof-of-concept for a new architecture, with RAN aiming at providing the model a higher level understanding of sequence patterns. To our best knowledge, we are the first to propose such an architecture. Out of the standard 85 UCR data sets, we have achieved 35 state-of-the-art results with 10 results matching current state-of-the-art results without further model fine-tuning. The results indicate that such architecture is promising in complex, long-sequence modeling and may have vast, cross-domain applications.
References in corpus (8)
- An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
- LSTM Fully Convolutional Networks for Time Series Classification
- Multivariate LSTM-FCNs for Time Series Classification
- Insights into LSTM Fully Convolutional Networks for Time Series Classification
- Linked Dynamic Graph CNN: Learning on Point Cloud via Linking Hierarchical Features
- Weighted Transformer Network for Machine Translation
- Deep Neural Network Ensembles for Time Series Classification
- Financial series prediction using Attention LSTM