Showing 2026Show all
3 papers · 1 filter
cs.LG2026
Learned Subspace Compression for Communication-Efficient Pipeline Parallelism
Paul Janson, Edouard Oyallon, Eugene Belilovsky
Pipeline parallelism enables training of large language models that exceed single-device memory, yet inter-stage activation communication becomes the dominant bottleneck when train…
cs.LG2026
Stabilizing Native Low-Rank LLM Pretraining
Paul Janson, Edouard Oyallon, Eugene Belilovsky
Foundation models have achieved remarkable success, yet their growing parameter counts pose significant computational and memory challenges. Low-rank factorization offers a promisi…
cs.LG2026
Heterogeneous Low-Bandwidth Pre-Training of LLMs
Yazan Obeidi, Amir Sarfi, Joel Lidin +2
Pre-training large language models (LLMs) increasingly requires distributed compute, yet bandwidth constraints make it difficult to scale beyond well-provisioned datacenters-especi…