Tensor Canonical Correlation Analysis with Convergence and Statistical Guarantees

Abstract

In many applications, such as classification of images or videos, it is of interest to develop a framework for tensor data instead of an ad-hoc way of transforming data to vectors due to the computational and under-sampling issues. In this paper, we study convergence and statistical properties of two-dimensional canonical correlation analysis \citep{Lee2007Two} under an assumption that data come from a probabilistic model. We show that carefully initialized the power method converges to the optimum and provide a finite sample bound. Then we extend this framework to tensor-valued data and propose the higher-order power method, which is commonly used in tensor decomposition, to extract the canonical directions. Our method can be used effectively in a large-scale data setting by solving the inner least squares problem with a stochastic gradient descent, and we justify convergence via the theory of Lojasiewicz's inequalities without any assumption on data generating process and initialization. For practical applications, we further develop (a) an inexact updating scheme which allows us to use the state-of-the-art stochastic gradient descent algorithm, (b) an effective initialization scheme which alleviates the problem of local optimum in non-convex optimization, and (c) a deflation procedure for extracting several canonical components. Empirical analyses on challenging data including gene expression and air pollution indexes in Taiwan, show the effectiveness and efficiency of the proposed methodology. Our results fill a missing, but crucial, part in the literature on tensor data.

References in corpus (21)

• Regularized Matrix Regression
• Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees
• Deep Variational Canonical Correlation Analysis
• The Noisy Power Method: A Meta Algorithm with Applications
• Randomized Nonlinear Component Analysis
• Nonparametric Canonical Correlation Analysis
• Scalable and Flexible Multiview MAX-VAR Canonical Correlation Analysis
• Independent component analysis for tensor-valued data
• A new convergence proof for the higher-order power method and generalizations
• Finding Linear Structure in Large Datasets with Scalable Canonical Correlation Analysis
• On the Convergence of Alternating Least Squares Optimisation in Tensor Format Representations
• Stochastic Approximation for Canonical Correlation Analysis
• Adaptive Canonical Correlation Analysis Based On Matrix Manifolds
• Stochastic Canonical Correlation Analysis
• Quadratic Optimization with Orthogonality Constraints: Explicit Lojasiewicz Exponent and Linear Convergence of Line-Search Methods
• Efficient coordinate-wise leading eigenvector computation
• Convergence of Alternating Least Squares Optimisation for Rank-One Approximation to High Order Tensors
• Factor Models for High-Dimensional Dynamic Networks: with Application to International Trade Flow Time Series 1981-2015
• Learning Influence-Receptivity Network Structure with Guarantee
• Provable Gaussian Embedding with One Observation
• Asymmetrically Weighted CCA And Hierarchical Kernel Sentence Embedding For Image & Text Retrieval