36 citations · 78 across the 16 of their papers we have counts for
13 papers · 1 filter
How Good is a Single Basin?
Kai Lion, Lorenzo Noci, Thomas Hofmann +1
The multi-modal nature of neural loss landscapes is often considered to be the main driver behind the empirical success of deep ensembles. In this work, we probe this belief by con…
Disentangling Linear Mode-Connectivity
Gul Sena Altintas, Gregor Bachmann, Lorenzo Noci +1
Linear mode-connectivity (LMC) (or lack thereof) is one of the intriguing characteristics of neural network loss landscapes. While empirically well established, it unfortunately st…
Navigating Scaling Laws: Compute Optimality in Adaptive Model Training
Sotiris Anagnostidis, Gregor Bachmann, Imanol Schlag +1
In recent years, the state-of-the-art in deep learning has been dominated by very large models that have been pre-trained on vast amounts of data. The paradigm is very simple: inve…
Scaling MLPs: A Tale of Inductive Bias
Gregor Bachmann, Sotiris Anagnostidis, Thomas Hofmann
In this work we revisit the most fundamental building block in deep learning, the multi-layer perceptron (MLP), and study the limits of its performance on vision tasks. Empirical i…
Random Teachers are Good Teachers
Felix Sarnthein, Gregor Bachmann, Sotiris Anagnostidis +1
In this work, we investigate the implicit regularization induced by teacher-student learning dynamics in self-distillation. To isolate its effect, we describe a simple experiment w…
The Curious Case of Benign Memorization
Sotiris Anagnostidis, Gregor Bachmann, Lorenzo Noci +1
Despite the empirical advances of deep learning across a variety of learning tasks, our theoretical understanding of its success is still very restricted. One of the key challenges…
How Tempering Fixes Data Augmentation in Bayesian Neural Networks
Gregor Bachmann, Lorenzo Noci, Thomas Hofmann
While Bayesian neural networks (BNNs) provide a sound and principled alternative to standard neural networks, an artificial sharpening of the posterior usually needs to be applied…
Generalization Through The Lens Of Leave-One-Out Error
Gregor Bachmann, Thomas Hofmann, Aurélien Lucchi
Despite the tremendous empirical success of deep learning models to solve various learning tasks, our theoretical understanding of their generalization ability is very limited. Cla…
Analytic Insights into Structure and Rank of Neural Network Hessian Maps
Sidak Pal Singh, Gregor Bachmann, Thomas Hofmann
The Hessian of a neural network captures parameter interactions through second-order derivatives of the loss. It is a fundamental object of study, closely tied to various problems…
Precise characterization of the prior predictive distribution of deep ReLU networks
Lorenzo Noci, Gregor Bachmann, Kevin Roth +2
Recent works on Bayesian neural networks (BNNs) have highlighted the need to better understand the implications of using Gaussian priors in combination with the compositional struc…
Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect
Lorenzo Noci, Kevin Roth, Gregor Bachmann +2
The "cold posterior effect" (CPE) in Bayesian deep learning describes the uncomforting observation that the predictive performance of Bayesian neural networks can be significantly…
Uniform Convergence, Adversarial Spheres and a Simple Remedy
Gregor Bachmann, Seyed-Mohsen Moosavi-Dezfooli, Thomas Hofmann
Previous work has cast doubt on the general framework of uniform convergence and its ability to explain generalization in neural networks. By considering a specific dataset, it was…
Constant Curvature Graph Convolutional Networks
Gregor Bachmann, Gary Bécigneul, Octavian-Eugen Ganea
Interest has been rising lately towards methods representing data in non-Euclidean spaces, e.g. hyperbolic or spherical, that provide specific inductive biases useful for certain r…