12 citations · 50 across the 49 of their papers we have counts for
10 papers · 1 filter
Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks
MohammadReza Davari, Eugene Belilovsky
The rapid development of AI systems has been greatly influenced by the emergence of foundation models. A common approach for targeted problems involves fine-tuning these pre-traine…
Meta-learning Optimizers for Communication-Efficient Learning
Charles-Étienne Joseph, Benjamin Thérien, Abhinav Moudgil +2
Communication-efficient variants of SGD, specifically local SGD, have received a great deal of interest in recent years. These approaches compute multiple gradient steps locally on…
Continual Pre-Training of Large Language Models: How to (re)warm your model?
Kshitij Gupta, Benjamin Thérien, Adam Ibrahim +5
Large language models (LLMs) are routinely pre-trained on billions of tokens, only to restart the process over again once new data becomes available. A much cheaper and more effici…
Adversarial Attacks on the Interpretation of Neuron Activation Maximization
Geraldin Nanfack, Alexander Fulleringer, Jonathan Marty +2
The internal functional behavior of trained Deep Neural Networks is notoriously difficult to interpret. Activation-maximization approaches are one set of techniques used to interpr…
Can Forward Gradient Match Backpropagation?
Louis Fournier, Stéphane Rivaud, Eugene Belilovsky +2
Forward Gradients - the idea of using directional derivatives in forward differentiation mode - have recently been shown to be utilizable for neural network training while avoiding…
: Accelerating Asynchronous Communication in Decentralized Deep Learning
Adel Nabli, Eugene Belilovsky, Edouard Oyallon
Distributed training of Deep Learning models has been critical to many recent successes in the field. Current standard methods primarily rely on synchronous centralized algorithms…
Guiding The Last Layer in Federated Learning with Pre-Trained Models
Gwen Legate, Nicolas Bernier, Lucas Caccia +2
Federated Learning (FL) is an emerging paradigm that allows a model to be trained across a number of participants without sharing data. Recent works have begun to consider the effe…
Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learning
Gwen Legate, Lucas Caccia, Eugene Belilovsky
In Federated Learning, a global model is learned by aggregating model updates computed at a set of independent client nodes, to reduce communication costs multiple gradient steps a…
Imitation from Observation With Bootstrapped Contrastive Learning
Medric Sonwa, Johanna Hansen, Eugene Belilovsky
Imitation from observation (IfO) is a learning paradigm that consists of training autonomous agents in a Markov Decision Process (MDP) by observing expert demonstrations without ac…
Local Learning with Neuron Groups
Adeetya Patel, Michael Eickenberg, Eugene Belilovsky
Traditional deep network training methods optimize a monolithic objective function jointly for all the components. This can lead to various inefficiencies in terms of potential par…