12 citations · 50 across the 49 of their papers we have counts for
10 papers · 2 filters
Non-Uniform Parameter-Wise Model Merging
Albert Manuel Orozco Camacho, Stefan Horoi, Guy Wolf +1
Combining multiple machine learning models has long been a technique for enhancing performance, particularly in distributed settings. Traditional approaches, such as model ensemble…
Not Only the Last-Layer Features for Spurious Correlations: All Layer Deep Feature Reweighting
Humza Wajid Hameed, Geraldin Nanfack, Eugene Belilovsky
Spurious correlations are a major source of errors for machine learning models, in particular when aiming for group-level fairness. It has been recently shown that a powerful appro…
Accelerating Training with Neuron Interaction and Nowcasting Networks
Boris Knyazev, Abhinav Moudgil, Guillaume Lajoie +2
Neural network training can be accelerated when a learnable update rule is used in lieu of classic adaptive optimizers (e.g. Adam). However, learnable update rules can be costly an…
Harmony in Diversity: Merging Neural Networks with Canonical Correlation Analysis
Stefan Horoi, Albert Manuel Orozco Camacho, Eugene Belilovsky +1
Combining the predictions of multiple trained models through ensembling is generally a good way to improve accuracy by leveraging the different learned features of the models, howe…
ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
Adel Nabli, Louis Fournier, Pierre Erbacher +3
Training LLMs relies on distributed implementations using multiple GPUs to compute gradients in parallel with sharded optimizers. However, synchronizing gradients in data parallel…
PETRA: Parallel End-to-end Training with Reversible Architectures
Stéphane Rivaud, Louis Fournier, Thomas Pumir +3
Reversible architectures have been shown to be capable of performing on par with their non-reversible architectures, being applied in deep learning for memory savings and generativ…
LO: Compute-Efficient Meta-Generalization of Learned Optimizers
Benjamin Thérien, Charles-Étienne Joseph, Boris Knyazev +3
Learned optimizers (LOs) have the potential to significantly reduce the wall-clock training time of neural networks. However, they can struggle to optimize unseen tasks (meta-gener…
WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average
Louis Fournier, Adel Nabli, Masih Aminbeidokhti +3
The performance of deep neural networks is enhanced by ensemble methods, which average the output of several models. However, this comes at an increased cost at inference. Weight a…
AdaFisher: Adaptive Second Order Optimization via Fisher Information
Damien Martins Gomes, Yanlei Zhang, Eugene Belilovsky +2
First-order optimization methods are currently the mainstream in training deep neural networks (DNNs). Optimizers like Adam incorporate limited curvature information by employing t…
Simple and Scalable Strategies to Continually Pre-train Large Language Models
Adam Ibrahim, Benjamin Thérien, Kshitij Gupta +5
Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes available. A much more efficient solution i…