Attention-based PCA

Rodrigo Maulen-Soto, Claire Boyer

We study attention mechanisms through the lens of a canonical unsupervised problem: principal component analysis (PCA). We show that, when trained on Gaussian data, both softmax an…