249 citations · 699 across the 39 of their papers we have counts for
6 papers · 1 filter
How Inner Optimizers Shape Replica Scaling
Eugene Belilovsky, Stefan Horoi, Edouard Oyallon +1
Training large language models (LLMs) across geographically distributed and poorly interconnected compute pools or data centers has become a topic of significant interest. This has…
Unifying Local Communications and Local Updates for LLM Pretraining
Pietro Cagnasso, Eugene Belilovsky, Edouard Oyallon
Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data centers, and lower-bandwidth links. Many pract…
Learned Subspace Compression for Communication-Efficient Pipeline Parallelism
Paul Janson, Edouard Oyallon, Eugene Belilovsky
Pipeline parallelism enables training of large language models that exceed single-device memory, yet inter-stage activation communication becomes the dominant bottleneck when train…
Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation
Killian Bakong, Laurent Massoulié, Edouard Oyallon +1
In this work we introduce methods to reduce the computational and memory costs of training deep neural networks. Our approach consists in replacing exact vector-jacobian products b…
Stabilizing Native Low-Rank LLM Pretraining
Paul Janson, Edouard Oyallon, Eugene Belilovsky
Foundation models have achieved remarkable success, yet their growing parameter counts pose significant computational and memory challenges. Low-rank factorization offers a promisi…
Test-time Generalization for Physics through Neural Operator Splitting
Louis Serrano, Jiequn Han, Edouard Oyallon +2
Neural operators have shown promise in learning solution maps of partial differential equations (PDEs), but they often struggle to generalize when test inputs lie outside the train…