EigenGame: PCA as a Nash Equilibrium
arXiv:2010.00554
Abstract
We present a novel view on principal component analysis (PCA) as a competitive game in which each approximate eigenvector is controlled by a player whose goal is to maximize their own utility function. We analyze the properties of this PCA game and the behavior of its gradient based updates. The resulting algorithm -- which combines elements from Oja's rule with a generalized Gram-Schmidt orthogonalization -- is naturally decentralized and hence parallelizable through message passing. We demonstrate the scalability of the algorithm with experiments on large image datasets and neural network activations. We discuss how this new view of PCA as a differentiable game can lead to further algorithmic developments and insights.
arXiv commentPublished as a conference paper at International Conference on Learning Representations (ICLR) 2021
References in corpus (8)
- Spectral Normalization for Generative Adversarial Networks
- Towards Federated Learning at Scale: System Design
- Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
- Global rates of convergence for nonconvex optimization on manifolds
- Natural Neural Networks
- Eigenvalue and Generalized Eigenvalue Problems: Tutorial
- Distributed Stochastic Algorithms for High-rate Streaming Principal Component Analysis
- Scalable Adaptive Stochastic Optimization Using Random Projections