37 citations · 63 across the 22 of their papers we have counts for
6 papers · 1 filter
Theoretical characterisation of the Gauss-Newton conditioning in Neural Networks
Jim Zhao, Sidak Pal Singh, Aurelien Lucchi
The Gauss-Newton (GN) matrix plays an important role in machine learning, most evident in its use as a preconditioning matrix for a wide family of popular adaptive methods to speed…
What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis
Weronika Ormaniec, Felix Dangel, Sidak Pal Singh
The Transformer architecture has inarguably revolutionized deep learning, overtaking classical architectures like multi-layer perceptrons (MLPs) and convolutional neural networks (…
Local vs Global continual learning
Giulia Lanzillotta, Sidak Pal Singh, Benjamin F. Grewe +1
Continual learning is the problem of integrating new information in a model while retaining the knowledge acquired in the past. Despite the tangible improvements achieved in recent…
Landscaping Linear Mode Connectivity
Sidak Pal Singh, Linara Adilova, Michael Kamp +3
The presence of linear paths in parameter space between two different network solutions in certain cases, i.e., linear mode connectivity (LMC), has garnered interest from both theo…
Hallmarks of Optimization Trajectories in Neural Networks: Directional Exploration and Redundancy
Sidak Pal Singh, Bobby He, Thomas Hofmann +1
We propose a fresh take on understanding the mechanisms of neural networks by analyzing the rich directional structure of optimization trajectories, represented by their pointwise…
Towards Meta-Pruning via Optimal Transport
Alexander Theus, Olin Geimer, Friedrich Wicke +3
Structural pruning of neural networks conventionally relies on identifying and discarding less important neurons, a practice often resulting in significant accuracy loss that neces…