135 citations · 182 across the 17 of their papers we have counts for
9 papers · 1 filter
Harnessing Synthetic Datasets: The Role of Shape Bias in Deep Neural Network Generalization
Elior Benarous, Sotiris Anagnostidis, Luca Biggio +1
Recent advancements in deep learning have been primarily driven by the use of large models trained on increasingly vast datasets. While neural scaling laws have emerged to predict…
Navigating Scaling Laws: Compute Optimality in Adaptive Model Training
Sotiris Anagnostidis, Gregor Bachmann, Imanol Schlag +1
In recent years, the state-of-the-art in deep learning has been dominated by very large models that have been pre-trained on vast amounts of data. The paradigm is very simple: inve…
Transformer Fusion with Optimal Transport
Moritz Imfeld, Jacopo Graldi, Marco Giordano +3
Fusion is a technique for merging multiple independently-trained neural networks in order to combine their capabilities. Past attempts have been restricted to the case of fully-con…
Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes
Alexandros Delitzas, Maria Parelli, Nikolas Hars +4
Training models to apply common-sense linguistic knowledge and visual concepts from 2D images to 3D scene understanding is a promising direction that researchers have only recently…
Scaling MLPs: A Tale of Inductive Bias
Gregor Bachmann, Sotiris Anagnostidis, Thomas Hofmann
In this work we revisit the most fundamental building block in deep learning, the multi-layer perceptron (MLP), and study the limits of its performance on vision tasks. Empirical i…
Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers
Sotiris Anagnostidis, Dario Pavllo, Luca Biggio +3
Autoregressive Transformers adopted in Large Language Models (LLMs) are hard to scale to long sequences. Despite several works trying to reduce their computational cost, most of LL…
CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes
Maria Parelli, Alexandros Delitzas, Nikolas Hars +4
Training models to apply linguistic knowledge and visual concepts from 2D images to 3D world understanding is a promising direction that researchers have only recently started to e…
OpenAssistant Conversations -- Democratizing Large Language Model Alignment
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte +15
Aligning large language models (LLMs) with human preferences has proven to drastically improve usability and has driven rapid adoption as demonstrated by ChatGPT. Alignment techniq…
Random Teachers are Good Teachers
Felix Sarnthein, Gregor Bachmann, Sotiris Anagnostidis +1
In this work, we investigate the implicit regularization induced by teacher-student learning dynamics in self-distillation. To isolate its effect, we describe a simple experiment w…