On Universal Features for High-Dimensional Learning and Inference

Abstract

We consider the problem of identifying universal low-dimensional features from high-dimensional data for inference tasks in settings involving learning. For such problems, we introduce natural notions of universality and we show a local equivalence among them. Our analysis is naturally expressed via information geometry, and represents a conceptually and computationally useful analysis. The development reveals the complementary roles of the singular value decomposition, Hirschfeld-Gebelein-Rényi maximal correlation, the canonical correlation and principle component analyses of Hotelling and Pearson, Tishby's information bottleneck, Wyner's common information, Ky Fan $k$-norms, and Brieman and Friedman's alternating conditional expectations algorithm. We further illustrate how this framework facilitates understanding and optimizing aspects of learning systems, including multinomial logistic (softmax) regression and the associated neural network architecture, matrix factorization methods for collaborative filtering and other applications, rank-constrained multivariate linear regression, and forms of semi-supervised learning.

References in corpus (10)

• Efficient Estimation of Word Representations in Vector Space
• Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization
• Opening the Black Box of Deep Neural Networks via Information
• A kernel method for canonical correlation analysis
• Robust Subspace Learning: Robust PCA, Robust Subspace Tracking, and Robust Subspace Recovery
• Neural Network Matrix Factorization
• Convex Sparse Matrix Factorizations
• Nonparametric Canonical Correlation Analysis
• Linear Bounds between Contraction Coefficients for $f$-Divergences
• Hypercontractivity and its applications

Cited by in corpus (2)

• Explainable, Stable, and Scalable Graph Convolutional Networks for Learning Graph Representation
• The Age of Correlated Features in Supervised Learning based Forecasting