12 citations · 50 across the 49 of their papers we have counts for
14 papers · 1 filter
Optimal Hydra: Model Heterogeneous Federated Learning for LLMs using Data-Informed Hydra-Nets
Gwen Legate, Rahaf Aljundi, Eugene Belilovsky
Large language models (LLMs) trained on vast amounts of data have demonstrated impressive performance over a range of tasks, but future performance gains will require increasing am…
How Inner Optimizers Shape Replica Scaling
Eugene Belilovsky, Stefan Horoi, Edouard Oyallon +1
Training large language models (LLMs) across geographically distributed and poorly interconnected compute pools or data centers has become a topic of significant interest. This has…
RL Post-Training Diffusion Language Models Along Their Own Generation Paths
Vaibhav Singh, Eugene Belilovsky
Diffusion language models are a promising and more efficient alternative to autoregressive models. However, most RL post-training techniques are not easily adapted to these models.…
Circuit Discovery Helps Detect LLM Jailbreaking: A Mechanistic Interpretability Study
Paria Mehrbod, Boris Knyazev, Guy Wolf +2
Despite extensive safety alignment, large language models (LLMs) remain vulnerable to jailbreak attacks that bypass safeguards to elicit harmful content. While prior work attribute…
Efficient Long-Horizon Learning for Learned Optimization
Xiaolong Huang, Benjamin Thérien, James Harrison +1
Learned optimization aims to improve upon hand-designed optimizers (e.g., Adam and Muon) by meta-learning small neural network optimizers over a distribution of tasks. While recent…
Can Model Merging Improve Aggregation in DiLoCo?
Stefan Horoi, Benjamin Thérien, Guy Wolf +1
Model merging techniques, which aggregate independently finetuned models into one to combine their capabilities, have become a topic of significant interest in recent years, with a…
Unifying Local Communications and Local Updates for LLM Pretraining
Pietro Cagnasso, Eugene Belilovsky, Edouard Oyallon
Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data centers, and lower-bandwidth links. Many pract…
Learned Subspace Compression for Communication-Efficient Pipeline Parallelism
Paul Janson, Edouard Oyallon, Eugene Belilovsky
Pipeline parallelism enables training of large language models that exceed single-device memory, yet inter-stage activation communication becomes the dominant bottleneck when train…
Path-independent Flow Matching for Multi-parameter Generative Dynamics
Francisco Téllez, AmirHossein Zamani, Philippe Martin +5
Flow Matching is a powerful framework for learning transport maps between probability distributions. Yet its standard single-parameter formulation is not designed to capture multi-…
Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet
Joel Lidin, Amir Sarfi, Erfan Miahi +6
Recently, there has been increased interest in globally distributed training, which has the promise to both reduce training costs and democratize participation in building large-sc…
Efficient Refusal Ablation in LLM through Optimal Transport
Geraldin Nanfack, Eugene Belilovsky, Elvis Dohmatob
Safety-aligned language models refuse harmful requests through learned refusal behaviors encoded in their internal representations. Recent activation-based jailbreaking methods cir…
Celo2: Towards Learned Optimization Free Lunch
Abhinav Moudgil, Boris Knyazev, Eugene Belilovsky
Learned optimizers are powerful alternatives to hand-designed update rules like Adam, yet they have seen limited practical adoption since they often fail to meta-generalize beyond…
Stabilizing Native Low-Rank LLM Pretraining
Paul Janson, Edouard Oyallon, Eugene Belilovsky
Foundation models have achieved remarkable success, yet their growing parameter counts pose significant computational and memory challenges. Low-rank factorization offers a promisi…
Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL
Erfan Miahi, Eugene Belilovsky
Bandwidth-constrained distributed reinforcement learning (RL) post-training of large language models is bottlenecked by two channels: weight synchronization from trainers to infere…