6 papers · 1 filter
Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
Peter SúkenÃk, Cristina López Amado, Christoph H. Lampert +1
This paper studies the role of sinks and diagonal patterns as attention switch and anti-oversmoothing mechanisms. We analyze geometric conditions under which sinks can be represent…
Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers
Peter SúkenÃk, Christoph H. Lampert, Marco Mondelli
The empirical emergence of neural collapse -- a surprising symmetry in the feature representations of the training data in the penultimate layer of deep neural networks -- has spur…
Average gradient outer product as a mechanism for deep neural collapse
Daniel Beaglehole, Peter SúkenÃk, Marco Mondelli +1
Deep Neural Collapse (DNC) refers to the surprisingly rigid structure of the data representations in the final layers of Deep Neural Networks (DNNs). Though the phenomenon has been…
Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?
Peter SúkenÃk, Marco Mondelli, Christoph Lampert
Deep neural networks (DNNs) exhibit a surprising structure in their final layer known as neural collapse (NC), and a growing body of works has currently investigated the propagatio…
Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse
Arthur Jacot, Peter SúkenÃk, Zihan Wang +1
Deep neural networks (DNNs) at convergence consistently represent the training data in the last layer via a highly symmetric geometric structure referred to as neural collapse. Thi…
Intriguing Properties of Input-dependent Randomized Smoothing
Peter SúkenÃk, Aleksei Kuvshinov, Stephan Günnemann
Randomized smoothing is currently considered the state-of-the-art method to obtain certifiably robust classifiers. Despite its remarkable performance, the method is associated with…