From the 1 of 20 linked papers with an AI index.
20 papers
The Gate Always Closes: On Injecting Auxiliary Signals into Frozen Vision-Language Models
Moshiur Farazi, Sameera Ramasinghe, Bekir Sait Ciftler +2
Auxiliary signal pathways in VLMs are routinely fitted with learnable gates so the optimiser can decide how much of the signal to admit. We find that the optimiser almost always de…
Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models
Gil Avraham, Violetta Shevchenko, Hadi Mohaghegh Dolatabadi +9
Agora is a system that enables large language models to be trained collectively across many heterogeneous, internet‑connected GPUs, using pipeline‑parallel sharding and fault‑toler…
Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
Sameera Ramasinghe, Thalaiyasingam Ajanthan, Gil Avraham +2
Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks. While…
Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo
Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan, Sameera Ramasinghe +7
To make large-scale distributed training practical outside high-bandwidth datacenters, we must reduce blocking, high-volume synchronization. While DiLoCo communicates infrequently,…
Taming Curvature: Architecture Warm-Up for Stable Transformer Training
Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi +6
Training billion-parameter Transformers is often brittle, with transient loss spikes and divergence that waste compute. Even though the recently developed Edge of Stability (EoS) t…
Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training
Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi +5
Pretraining language models with extended context windows enhances their ability to leverage rich information during generation. Existing methods split input sequences into chunks,…