2.1k citations · 5.2k across the 53 of their papers we have counts for
3 papers · 2 filters
Scaling MLPs: A Tale of Inductive Bias
Gregor Bachmann, Sotiris Anagnostidis, Thomas Hofmann
In this work we revisit the most fundamental building block in deep learning, the multi-layer perceptron (MLP), and study the limits of its performance on vision tasks. Empirical i…
Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learning
Sanghwan Kim, Lorenzo Noci, Antonio Orvieto +1
In contrast to the natural capabilities of humans to learn new tasks in a sequential fashion, neural networks are known to suffer from catastrophic forgetting, where the model's pe…
Random Teachers are Good Teachers
Felix Sarnthein, Gregor Bachmann, Sotiris Anagnostidis +1
In this work, we investigate the implicit regularization induced by teacher-student learning dynamics in self-distillation. To isolate its effect, we describe a simple experiment w…