From the 1 of 11 linked papers with an AI index.
11 papers
Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models
Gil Avraham, Violetta Shevchenko, Hadi Mohaghegh Dolatabadi +9
Agora is a system that enables large language models to be trained collectively across many heterogeneous, internet‑connected GPUs, using pipeline‑parallel sharding and fault‑toler…
Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo
Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan, Sameera Ramasinghe +7
To make large-scale distributed training practical outside high-bandwidth datacenters, we must reduce blocking, high-volume synchronization. While DiLoCo communicates infrequently,…
Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training
Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi +5
Pretraining language models with extended context windows enhances their ability to leverage rich information during generation. Existing methods split input sequences into chunks,…
Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization
Alexander Long, Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan +5
We consider a decentralized setup in which the participants collaboratively train and serve a large neural network, and where each participant only processes a subset of the model.…
NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training
Hadi Mohaghegh Dolatabadi, Thalaiyasingam Ajanthan, Sameera Ramasinghe +7
The rapid progress of large language models (LLMs) is increasingly constrained by memory and deployment costs, motivating compression methods for practical deployment. Many state-o…
SENTINEL: Stagewise Integrity Verification for Pipeline Parallel Decentralized Training
Hadi Mohaghegh Dolatabadi, Thalaiyasingam Ajanthan, Sameera Ramasinghe +5
Decentralized training introduces critical security risks when executed across untrusted, geographically distributed nodes. While existing Byzantine-tolerant literature addresses d…