2 citations · 3 across the 10 of their papers we have counts for
11 papers
The Safety Operator: Modulating the Expression of Safety Instructions via Spectral Optimization
Benoit Dherin, Michael Munn, Xavier Gonzalvo +14
Context tokens in a transformer-based language model can be absorbed into the model's weights as a multiplicative operator. We study this operator in the setting of safety instruct…
Equivalence of Context and Parameter Updates in Modern Transformer Blocks
Adrian Goldwaser, Michael Munn, Javier Gonzalvo +1
Recent research has established that the impact of context in a vanilla transformer can be represented implicitly by forming a token-dependent, rank-1 patch to its MLP weights. Thi…
On residual network depth
Benoit Dherin, Michael Munn
Deep residual architectures, such as ResNet and the Transformer, have enabled models of unprecedented depth, yet a formal understanding of why depth is so effective remains an open…
Transmuting prompts into weights
Hanna Mazzawi, Benoit Dherin, Michael Munn +3
A growing body of research has demonstrated that the behavior of large language models can be effectively controlled at inference time by directly modifying their internal states,…
Learning without training: The implicit dynamics of in-context learning
Benoit Dherin, Michael Munn, Hanna Mazzawi +2
One of the most striking features of Large Language Models (LLMs) is their ability to learn in-context. Namely at inference time an LLM is able to learn new patterns without any ad…
Learning by solving differential equations
Benoit Dherin, Michael Munn, Hanna Mazzawi +3
Modern deep learning algorithms use variations of gradient descent as their main learning methods. Gradient descent can be understood as the simplest Ordinary Differential Equation…
How iteration order influences convergence and stability in deep learning
Benoit Dherin, Benny Avelin, Anders Karlsson +3
Despite exceptional achievements, training neural networks remains computationally expensive and is often plagued by instabilities that can degrade convergence. While learning rate…
A Bayesian Model Selection Criterion for Selecting Pretraining Checkpoints
Michael Munn, Susan Wei
Recent advances in artificial intelligence have been fueled by the development of foundation models such as BERT, GPT, T5, and Vision Transformers. These models are first pretraine…
A Margin-based Multiclass Generalization Bound via Geometric Complexity
Michael Munn, Benoit Dherin, Javier Gonzalvo
There has been considerable effort to better understand the generalization capabilities of deep neural networks both as a means to unlock a theoretical understanding of their succe…
The Impact of Geometric Complexity on Neural Collapse in Transfer Learning
Michael Munn, Benoit Dherin, Javier Gonzalvo
Many of the recent remarkable advances in computer vision and language models can be attributed to the success of transfer learning via the pre-training of large foundation models.…
Privacy-Preserving Serverless Edge Learning with Decentralized Small Data
Shih-Chun Lin, Chia-Hung Lin
In the last decade, data-driven algorithms outperformed traditional optimization-based algorithms in many research areas, such as computer vision, natural language processing, etc.…