6 papers
Model Fusion via Retrofitting
Phoomraphee Luenam, Andreas Spanopoulos, Amit Sant +3
Model fusion seeks to combine independently trained neural networks into a single model without retraining, but is complicated by representational divergence arising from permutati…
Generalized Linear Mode Connectivity for Transformers
Alexander Theus, Alessandro Cabodi, Sotiris Anagnostidis +3
Understanding the geometry of neural network loss landscapes is a central question in deep learning, with implications for generalization and optimization. A striking phenomenon is…
Accelerating Neural Network Training Along Sharp and Flat Directions
Daniyar Zakarin, Sidak Pal Singh
Recent work has highlighted a surprising alignment between gradients and the top eigenspace of the Hessian -- termed the Dominant subspace -- during neural network training. Concur…
What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis
Weronika Ormaniec, Felix Dangel, Sidak Pal Singh
The Transformer architecture has inarguably revolutionized deep learning, overtaking classical architectures like multi-layer perceptrons (MLPs) and convolutional neural networks (…
Theoretical characterisation of the Gauss-Newton conditioning in Neural Networks
Jim Zhao, Sidak Pal Singh, Aurelien Lucchi
The Gauss-Newton (GN) matrix plays an important role in machine learning, most evident in its use as a preconditioning matrix for a wide family of popular adaptive methods to speed…
Avoiding spurious sharpness minimization broadens applicability of SAM
Sidak Pal Singh, Hossein Mobahi, Atish Agarwala +1
Curvature regularization techniques like Sharpness Aware Minimization (SAM) have shown great promise in improving generalization on vision tasks. However, we find that SAM performs…