249 citations · 699 across the 39 of their papers we have counts for
41 papers
How Inner Optimizers Shape Replica Scaling
Eugene Belilovsky, Stefan Horoi, Edouard Oyallon +1
Training large language models (LLMs) across geographically distributed and poorly interconnected compute pools or data centers has become a topic of significant interest. This has…
Unifying Local Communications and Local Updates for LLM Pretraining
Pietro Cagnasso, Eugene Belilovsky, Edouard Oyallon
Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data centers, and lower-bandwidth links. Many pract…
Learned Subspace Compression for Communication-Efficient Pipeline Parallelism
Paul Janson, Edouard Oyallon, Eugene Belilovsky
Pipeline parallelism enables training of large language models that exceed single-device memory, yet inter-stage activation communication becomes the dominant bottleneck when train…
Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation
Killian Bakong, Laurent Massoulié, Edouard Oyallon +1
In this work we introduce methods to reduce the computational and memory costs of training deep neural networks. Our approach consists in replacing exact vector-jacobian products b…
Stabilizing Native Low-Rank LLM Pretraining
Paul Janson, Edouard Oyallon, Eugene Belilovsky
Foundation models have achieved remarkable success, yet their growing parameter counts pose significant computational and memory challenges. Low-rank factorization offers a promisi…
Test-time Generalization for Physics through Neural Operator Splitting
Louis Serrano, Jiequn Han, Edouard Oyallon +2
Neural operators have shown promise in learning solution maps of partial differential equations (PDEs), but they often struggle to generalize when test inputs lie outside the train…
Model Parallelism With Subnetwork Data Parallelism
Vaibhav Singh, Zafir Khalid, Pietro Cagnasso +2
Pre-training large neural networks at scale imposes heavy memory demands on accelerators and often requires costly communication. We introduce Subnetwork Data Parallelism (SDP), a…
DISCO: learning to DISCover an evolution Operator for multi-physics-agnostic prediction
Rudy Morel, Jiequn Han, Edouard Oyallon
We address the problem of predicting the next state of a dynamical system governed by unknown temporal partial differential equations (PDEs) using only a short trajectory. While st…
ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
Adel Nabli, Louis Fournier, Pierre Erbacher +3
Training LLMs relies on distributed implementations using multiple GPUs to compute gradients in parallel with sharded optimizers. However, synchronizing gradients in data parallel…
PETRA: Parallel End-to-end Training with Reversible Architectures
Stéphane Rivaud, Louis Fournier, Thomas Pumir +3
Reversible architectures have been shown to be capable of performing on par with their non-reversible architectures, being applied in deep learning for memory savings and generativ…
LO: Compute-Efficient Meta-Generalization of Learned Optimizers
Benjamin Thérien, Charles-Étienne Joseph, Boris Knyazev +3
Learned optimizers (LOs) have the potential to significantly reduce the wall-clock training time of neural networks. However, they can struggle to optimize unseen tasks (meta-gener…
WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average
Louis Fournier, Adel Nabli, Masih Aminbeidokhti +3
The performance of deep neural networks is enhanced by ensemble methods, which average the output of several models. However, this comes at an increased cost at inference. Weight a…
Cyclic Data Parallelism for Efficient Parallelism of Deep Neural Networks
Louis Fournier, Edouard Oyallon
Training large deep learning models requires parallelization techniques to scale. In existing methods such as Data Parallelism or ZeRO-DP, micro-batches of data are processed in pa…
Vectorizing string entries for data processing on tables: when are larger language models better?
Léo Grinsztajn, Edouard Oyallon, Myung Jun Kim +1
There are increasingly efficient data processing pipelines that work on vectors of numbers, for instance most machine learning models, or vector databases for fast similarity searc…
Can Forward Gradient Match Backpropagation?
Louis Fournier, Stéphane Rivaud, Eugene Belilovsky +2
Forward Gradients - the idea of using directional derivatives in forward differentiation mode - have recently been shown to be utilizable for neural network training while avoiding…
: Accelerating Asynchronous Communication in Decentralized Deep Learning
Adel Nabli, Eugene Belilovsky, Edouard Oyallon
Distributed training of Deep Learning models has been critical to many recent successes in the field. Current standard methods primarily rely on synchronous centralized algorithms…
Guiding The Last Layer in Federated Learning with Pre-Trained Models
Gwen Legate, Nicolas Bernier, Lucas Caccia +2
Federated Learning (FL) is an emerging paradigm that allows a model to be trained across a number of participants without sharing data. Recent works have begun to consider the effe…
DADAO: Decoupled Accelerated Decentralized Asynchronous Optimization
Adel Nabli, Edouard Oyallon
This work introduces DADAO: the first decentralized, accelerated, asynchronous, primal, first-order algorithm to minimize a sum of -smooth and -strongly convex functions dist…
Why do tree-based models still outperform deep learning on tabular data?
Léo Grinsztajn, Edouard Oyallon, Gaël Varoquaux
While deep learning has enabled tremendous progress on text and image datasets, its superiority on tabular data is not clear. We contribute extensive benchmarks of standard and nov…
On Non-Linear operators for Geometric Deep Learning
Grégoire Sergeant-Perthuis, Jakob Maier, Joan Bruna +1
This work studies operators mapping vector and scalar fields defined over a manifold , and which commute with its group of diffeomorphisms .…
Gradient Masked Averaging for Federated Learning
Irene Tenison, Sai Aravind Sreeramadas, Vaikkunth Mugunthan +3
Federated learning (FL) is an emerging paradigm that permits a large number of clients with heterogeneous data to coordinate learning of a unified global model without the need to…
Deep Reinforcement Learning for L3 Slice Localization in Sarcopenia Assessment
Othmane Laousy, Guillaume Chassagnon, Edouard Oyallon +3
Sarcopenia is a medical condition characterized by a reduction in muscle mass and function. A quantitative diagnosis technique consists of localizing the CT slice passing through t…
Decoupled Greedy Learning of CNNs for Synchronous and Asynchronous Distributed Learning
Eugene Belilovsky, Louis Leconte, Lucas Caccia +2
A commonly cited inefficiency of neural network training using back-propagation is the update locking problem: each layer must wait for the signal to propagate through the full net…
Low-Rank Projections of GCNs Laplacian
Nathan Grinsztajn, Philippe Preux, Edouard Oyallon
In this work, we study the behavior of standard models for community detection under spectral manipulations. Through various ablation experiments, we evaluate the impact of bandpas…