Tensor-Train Networks for Learning Predictive Modeling of Multidimensional Data

Abstract

In this work, we firstly apply the Train-Tensor (TT) networks to construct a compact representation of the classical Multilayer Perceptron, representing a reduction of up to 95% of the coefficients. A comparative analysis between tensor model and standard multilayer neural networks is also carried out in the context of prediction of the Mackey-Glass noisy chaotic time series and NASDAQ index. We show that the weights of a multidimensional regression model can be learned by means of TT network and the optimization of TT weights is a more robust to the impact of coefficient initialization and hyper-parameter setting. Furthermore, an efficient algorithm based on alternating least squares has been proposed for approximating the weights in TT-format with a reduction of computational calculus, providing a much faster convergence than the well-known adaptive learning-method algorithms, widely applied for optimizing neural networks.

arXiv comment34 pages, 16 figures

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Time Series Forecasting With Deep Learning: A Survey
• On the Convergence of Adam and Beyond
• Era of Big Data Processing: A New Approach via Tensor Networks and Tensor Decompositions
• Deep Multi-task Representation Learning: A Tensor Factorisation Approach
• Long-term Forecasting using Higher Order Tensor RNNs
• TensorNetwork for Machine Learning
• Generative Tensor Network Classification Model for Supervised Machine Learning
• Infinitely Wide Tensor Networks as Gaussian Process
• Tensor train decompositions on recurrent networks