ORCCA: Optimal Randomized Canonical Correlation Analysis

Abstract

Random features approach has been widely used for kernel approximation in large-scale machine learning. A number of recent studies have explored data-dependent sampling of features, modifying the stochastic oracle from which random features are sampled. While proposed techniques in this realm improve the approximation, their suitability is often verified on a single learning task. In this paper, we propose a task-specific scoring rule for selecting random features, which can be employed for different applications with some adjustments. We restrict our attention to Canonical Correlation Analysis (CCA), and we provide a novel, principled guide for finding the score function maximizing the canonical correlations. We prove that this method, called ORCCA, can outperform (in expectation) the corresponding Kernel CCA with a default kernel. Numerical experiments verify that ORCCA is significantly superior than other approximation techniques in the CCA task.

References in corpus (14)

• Generalization Properties of Learning with Random Features
• Orthogonal Random Features
• Deep Variational Canonical Correlation Analysis
• Random Fourier Features for Kernel Ridge Regression: Approximation Bounds and Statistical Guarantees
• Nonparametric Canonical Correlation Analysis
• Kernel Approximation Methods for Speech Recognition
• Bayesian Nonparametric Kernel-Learning
• Large-Scale Approximate Kernel Canonical Correlation Analysis
• Compact Nonlinear Maps and Circulant Extensions
• But How Does It Work in Theory? Linear SVM with Random Features
• Stochastic Approximation for Canonical Correlation Analysis
• Data-dependent compression of random features for large-scale kernel approximation
• Decentralised Learning with Random Features and Distributed Gradient Descent
• On Sampling Random Features From Empirical Leverage Scores: Implementation and Theoretical Guarantees