37 citations · 63 across the 22 of their papers we have counts for
4 papers · 1 filter
Model Fusion via Retrofitting
Phoomraphee Luenam, Andreas Spanopoulos, Amit Sant +3
Model fusion seeks to combine independently trained neural networks into a single model without retraining, but is complicated by representational divergence arising from permutati…
Generalized Linear Mode Connectivity for Transformers
Alexander Theus, Alessandro Cabodi, Sotiris Anagnostidis +3
Understanding the geometry of neural network loss landscapes is a central question in deep learning, with implications for generalization and optimization. A striking phenomenon is…
Accelerating Neural Network Training Along Sharp and Flat Directions
Daniyar Zakarin, Sidak Pal Singh
Recent work has highlighted a surprising alignment between gradients and the top eigenspace of the Hessian -- termed the Dominant subspace -- during neural network training. Concur…
Avoiding spurious sharpness minimization broadens applicability of SAM
Sidak Pal Singh, Hossein Mobahi, Atish Agarwala +1
Curvature regularization techniques like Sharpness Aware Minimization (SAM) have shown great promise in improving generalization on vision tasks. However, we find that SAM performs…