4 papers
Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
Peter SúkenÃk, Cristina López Amado, Christoph H. Lampert +1
This paper studies the role of sinks and diagonal patterns as attention switch and anti-oversmoothing mechanisms. We analyze geometric conditions under which sinks can be represent…
Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers
Peter SúkenÃk, Christoph H. Lampert, Marco Mondelli
The empirical emergence of neural collapse -- a surprising symmetry in the feature representations of the training data in the penultimate layer of deep neural networks -- has spur…
Average gradient outer product as a mechanism for deep neural collapse
Daniel Beaglehole, Peter SúkenÃk, Marco Mondelli +1
Deep Neural Collapse (DNC) refers to the surprisingly rigid structure of the data representations in the final layers of Deep Neural Networks (DNNs). Though the phenomenon has been…
Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?
Peter SúkenÃk, Marco Mondelli, Christoph Lampert
Deep neural networks (DNNs) exhibit a surprising structure in their final layer known as neural collapse (NC), and a growing body of works has currently investigated the propagatio…