3 citations · 6 across the 8 of their papers we have counts for
9 papers
Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
Peter Súkeník, Cristina López Amado, Christoph H. Lampert +1
This paper studies the role of sinks and diagonal patterns as attention switch and anti-oversmoothing mechanisms. We analyze geometric conditions under which sinks can be represent…
Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers
Peter Súkeník, Christoph H. Lampert, Marco Mondelli
The empirical emergence of neural collapse -- a surprising symmetry in the feature representations of the training data in the penultimate layer of deep neural networks -- has spur…
Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse
Arthur Jacot, Peter Súkeník, Zihan Wang +1
Deep neural networks (DNNs) at convergence consistently represent the training data in the last layer via a highly symmetric geometric structure referred to as neural collapse. Thi…
Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?
Peter Súkeník, Marco Mondelli, Christoph Lampert
Deep neural networks (DNNs) exhibit a surprising structure in their final layer known as neural collapse (NC), and a growing body of works has currently investigated the propagatio…
Average gradient outer product as a mechanism for deep neural collapse
Daniel Beaglehole, Peter Súkeník, Marco Mondelli +1
Deep Neural Collapse (DNC) refers to the surprisingly rigid structure of the data representations in the final layers of Deep Neural Networks (DNNs). Though the phenomenon has been…
Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model
Peter Súkeník, Marco Mondelli, Christoph Lampert
Neural collapse (NC) refers to the surprising structure of the last layer of deep neural networks in the terminal phase of gradient descent training. Recently, an increasing amount…
The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes
Peter Kocsis, Peter Súkeník, Guillem Brasó +3
Convolutional neural networks were the standard for solving many computer vision tasks until recently, when Transformers of MLP-based architectures have started to show competitive…
Generalization In Multi-Objective Machine Learning
Peter Súkeník, Christoph H. Lampert
Modern machine learning tasks often require considering not just one but multiple objectives. For example, besides the prediction quality, this could be the efficiency, robustness…
Intriguing Properties of Input-dependent Randomized Smoothing
Peter Súkeník, Aleksei Kuvshinov, Stephan Günnemann
Randomized smoothing is currently considered the state-of-the-art method to obtain certifiably robust classifiers. Despite its remarkable performance, the method is associated with…