Large-Scale Approximate Kernel Canonical Correlation Analysis

Abstract

Kernel canonical correlation analysis (KCCA) is a nonlinear multi-view representation learning technique with broad applicability in statistics and machine learning. Although there is a closed-form solution for the KCCA objective, it involves solving an $N\times N$ eigenvalue system where $N$ is the training set size, making its computational requirements in both memory and time prohibitive for large-scale problems. Various approximation techniques have been developed for KCCA. A commonly used approach is to first transform the original inputs to an $M$-dimensional random feature space so that inner products in the feature space approximate kernel evaluations, and then apply linear CCA to the transformed inputs. In many applications, however, the dimensionality $M$ of the random feature space may need to be very large in order to obtain a sufficiently good approximation; it then becomes challenging to perform the linear CCA step on the resulting very high-dimensional data matrices. We show how to use a stochastic optimization algorithm, recently proposed for linear CCA and its neural-network extension, to further alleviate the computation requirements of approximate KCCA. This approach allows us to run approximate KCCA on a speech dataset with $1.4$ million training samples and a random feature space of dimensionality $M=100000$ on a typical workstation.

Published as a conference paper at International Conference on Learning Representations (ICLR) 2016

References in corpus (7)

• A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
• A kernel method for canonical correlation analysis
• Quasi-Monte Carlo Feature Maps for Shift-Invariant Kernels
• Finding Linear Structure in Large Datasets with Scalable Canonical Correlation Analysis
• Large scale canonical correlation analysis with iterative least squares
• Adaptive Canonical Correlation Analysis Based On Matrix Manifolds
• Stochastic Optimization for Deep CCA via Nonlinear Orthogonal Iterations

Cited by in corpus (7)

• Stochastic Canonical Correlation Analysis
• Efficient Algorithms for Large-scale Generalized Eigenvector Computation and Canonical Correlation Analysis
• Quantum-inspired canonical correlation analysis for exponentially large dimensional data
• Robust Matrix Elastic Net based Canonical Correlation Analysis: An Effective Algorithm for Multi-View Unsupervised Learning
• Deep Within-Class Covariance Analysis for Robust Audio Representation Learning
• ORCCA: Optimal Randomized Canonical Correlation Analysis
• Efficient Globally Convergent Stochastic Optimization for Canonical Correlation Analysis