13 citations · 42 across the 13 of their papers we have counts for
14 papers
The Safety Operator: Modulating the Expression of Safety Instructions via Spectral Optimization
Benoit Dherin, Michael Munn, Xavier Gonzalvo +14
Context tokens in a transformer-based language model can be absorbed into the model's weights as a multiplicative operator. We study this operator in the setting of safety instruct…
Grow, Don't Overwrite: Fine-tuning Without Forgetting
Dyah Adila, Hanna Mazzawi, Benoit Dherin +1
Adapting pre-trained models to specialized tasks often leads to catastrophic forgetting, where new knowledge overwrites foundational capabilities. Existing methods either compromis…
Transmuting prompts into weights
Hanna Mazzawi, Benoit Dherin, Michael Munn +3
A growing body of research has demonstrated that the behavior of large language models can be effectively controlled at inference time by directly modifying their internal states,…
Learning without training: The implicit dynamics of in-context learning
Benoit Dherin, Michael Munn, Hanna Mazzawi +2
One of the most striking features of Large Language Models (LLMs) is their ability to learn in-context. Namely at inference time an LLM is able to learn new patterns without any ad…
Learning by solving differential equations
Benoit Dherin, Michael Munn, Hanna Mazzawi +3
Modern deep learning algorithms use variations of gradient descent as their main learning methods. Gradient descent can be understood as the simplest Ordinary Differential Equation…
How iteration order influences convergence and stability in deep learning
Benoit Dherin, Benny Avelin, Anders Karlsson +3
Despite exceptional achievements, training neural networks remains computationally expensive and is often plagued by instabilities that can degrade convergence. While learning rate…
Majority Kernels: An Approach to Leverage Big Model Dynamics for Efficient Small Model Training
Hanna Mazzawi, Pranjal Awasthi, Xavi Gonzalvo +1
Recent breakthroughs and successful deployment of large language and vision models in a constrained environment predominantly follow a two phase approach. First, large models are t…
Deep Fusion: Efficient Network Training via Pre-trained Initializations
Hanna Mazzawi, Xavi Gonzalvo, Michael Wunder +2
In recent years, deep learning has made remarkable progress in a wide range of domains, with a particularly notable impact on natural language processing tasks. One of the challeng…
Towards Task and Architecture-Independent Generalization Gap Predictors
Scott Yak, Javier Gonzalvo, Hanna Mazzawi
Can we use deep learning to predict when deep learning works? Our results suggest the affirmative. We created a dataset by training 13,500 neural networks with different architectu…
AdaNet: A Scalable and Flexible Framework for Automatically Learning Ensembles
Charles Weill, Javier Gonzalvo, Vitaly Kuznetsov +9
AdaNet is a lightweight TensorFlow-based (Abadi et al., 2015) framework for automatically learning high-quality ensembles with minimal expert intervention. Our framework is inspire…
Improving Neural Architecture Search Image Classifiers via Ensemble Learning
Vladimir Macko, Charles Weill, Hanna Mazzawi +1
Finding the best neural network architecture requires significant time, resources, and human expertise. These challenges are partially addressed by neural architecture search (NAS)…
Non-Adaptive Learning a Hidden Hipergraph
Hasan Abasi, Nader H. Bshouty, Hanna Mazzawi
We give a new deterministic algorithm that non-adaptively learns a hidden hypergraph from edge-detecting queries. All previous non-adaptive algorithms either run in exponential tim…
On Exact Learning Monotone DNF from Membership Queries
Hasan Abasi, Nader H. Bshouty, Hanna Mazzawi
In this paper, we study the problem of learning a monotone DNF with at most terms of size (number of variables in each term) at most ( term -MDNF) from membership que…
Optimal Query Complexity for Reconstructing Hypergraphs
Nader H. Bshouty, Hanna Mazzawi
In this paper we consider the problem of reconstructing a hidden weighted hypergraph of constant rank using additive queries. We prove the following: Let be a weighted hidden h…