Stochastic Optimization for Deep CCA via Nonlinear Orthogonal Iterations

Abstract

Deep CCA is a recently proposed deep neural network extension to the traditional canonical correlation analysis (CCA), and has been successful for multi-view representation learning in several domains. However, stochastic optimization of the deep CCA objective is not straightforward, because it does not decouple over training examples. Previous optimizers for deep CCA are either batch-based algorithms or stochastic optimization using large minibatches, which can have high memory consumption. In this paper, we tackle the problem of stochastic optimization for deep CCA with small minibatches, based on an iterative solution to the CCA objective, and show that we can achieve as good performance as previous optimizers and thus alleviate the memory requirement.

in 2015 Annual Allerton Conference on Communication, Control and Computing

References in corpus (4)

• No More Pesky Learning Rates
• Finding Linear Structure in Large Datasets with Scalable Canonical Correlation Analysis
• Large scale canonical correlation analysis with iterative least squares
• Adaptive Canonical Correlation Analysis Based On Matrix Manifolds

Cited by in corpus (7)

• On Deep Multi-View Representation Learning: Objectives and Optimization
• Large-Scale Approximate Kernel Canonical Correlation Analysis
• Acoustic Feature Learning via Deep Variational Canonical Correlation Analysis
• DeMIAN: Deep Modality Invariant Adversarial Network
• Practical Cross-modal Manifold Alignment for Grounded Language
• A C++ library for Multimodal Deep Learning
• Application of multiview techniques to NHANES dataset