Spectral Learning on Matrices and Tensors

Abstract

Spectral methods have been the mainstay in several domains such as machine learning and scientific computing. They involve finding a certain kind of spectral decomposition to obtain basis functions that can capture important structures for the problem at hand. The most common spectral method is the principal component analysis (PCA). It utilizes the top eigenvectors of the data covariance matrix, e.g. to carry out dimensionality reduction. This data pre-processing step is often effective in separating signal from noise. PCA and other spectral techniques applied to matrices have several limitations. By limiting to only pairwise moments, they are effectively making a Gaussian approximation on the underlying data and fail on data with hidden variables which lead to non-Gaussianity. However, in most data sets, there are latent effects that cannot be directly observed, e.g., topics in a document corpus, or underlying causes of a disease. By extending the spectral decomposition methods to higher order moments, we demonstrate the ability to learn a wide range of latent variable models efficiently. Higher-order moments can be represented by tensors, and intuitively, they can encode more information than just pairwise moment matrices. More crucially, tensor decomposition can pick up latent effects that are missed by matrix methods, e.g. uniquely identify non-orthogonal components. Exploiting these aspects turns out to be fruitful for provable unsupervised learning of a wide range of latent variable models. We also outline the computational techniques to design efficient tensor decomposition methods. We introduce Tensorly, which has a simple python interface for expressing tensor operations. It has a flexible back-end system supporting NumPy, PyTorch, TensorFlow and MXNet amongst others, allowing multi-GPU and CPU operations and seamless integration with deep-learning functionalities.

References in corpus (21)

• Scikit-learn: Machine Learning in Python
• The NumPy array: a structure for efficient numerical computation
• Probabilistic Latent Semantic Analysis
• Tensor Decomposition for Signal Processing and Machine Learning
• Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization
• TensorLy: Tensor Learning in Python
• Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition
• A Spectral Algorithm for Latent Dirichlet Allocation
• Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
• What Regularized Auto-Encoders Learn from the Data Generating Distribution
• Sum-of-squares proofs and the quest toward optimal algorithms
• Tensor Regression Networks
• Density Estimation in Infinite Dimensional Exponential Families
• A Method of Moments for Mixture Models and Hidden Markov Models
• Decomposing Overcomplete 3rd Order Tensors using Sum-of-Squares Algorithms
• Score Function Features for Discriminative Learning: Matrix and Tensor Framework
• Identifiability and Unmixing of Latent Parse Trees
• Learning mixtures of spherical Gaussians: moment methods and spectral decompositions
• Exact tensor completion with sum-of-squares
• Sample Complexity Analysis for Learning Overcomplete Latent Variable Models through Tensor Methods
• Tensor Contraction Layers for Parsimonious Deep Nets

Cited by in corpus (5)

• Tensor Methods in Computer Vision and Deep Learning
• Computing linear sections of varieties: quantum entanglement, tensor decompositions and beyond
• Tensor decomposition for learning Gaussian mixtures from moments
• Incremental Bayesian tensor learning for structural monitoring data imputation and response forecasting
• Analyzing Political Text at Scale with Online Tensor LDA