Neural Collapse with Cross-Entropy Loss

Abstract

We consider the variational problem of cross-entropy loss with $n$ feature vectors on a unit hypersphere in ${\mathbb{R}}^d$. We prove that when $d\ge n-1$, the global minimum is given by the simplex equiangular tight frame, which justifies the neural collapse behavior. We also prove that as $n\to \infty$ with fixed $d$, the minimizing points will distribute uniformly on the hypersphere and show a connection with the frame potential of Benedetto & Fickus.

References in corpus (1)

• Prevalence of Neural Collapse during the terminal phase of deep learning training