10 citations · 15 across the 6 of their papers we have counts for
11 papers
Communication-Efficient LLM Adaptation over Decentralized GPU Meshes
Sameera Ramasinghe, Shamane Siriwardhana, Thalaiyasingam Ajanthan +8
Decentralized training enables large-model training over low-end GPUs and internet-grade connections, but communication along both data-parallel and pipeline-parallel axes becomes…
Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models
Gil Avraham, Violetta Shevchenko, Hadi Mohaghegh Dolatabadi +9
Training large language models at the multi-billion to trillion parameter scale is confined to datacenters, where data-parallel (DP) and model-parallel (MP) techniques presume homo…
Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo
Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan, Sameera Ramasinghe +7
To make large-scale distributed training practical outside high-bandwidth datacenters, we must reduce blocking, high-volume synchronization. While DiLoCo communicates infrequently,…
Merging in a Bottle: Differentiable Adaptive Merging (DAM) and the Path from Averaging to Automation
Thomas Gauthier-Caron, Shamane Siriwardhana, Elliot Stein +5
By merging models, AI systems can combine the distinct strengths of separate language models, achieving a balance between multiple capabilities without requiring substantial retrai…
Domain Adaptation of Llama3-70B-Instruct through Continual Pre-Training and Model Merging: A Comprehensive Evaluation
Shamane Siriwardhana, Mark McQuade, Thomas Gauthier +8
We conducted extensive experiments on domain adaptation of the Meta-Llama-3-70B-Instruct model on SEC data, exploring its performance on both general and domain-specific benchmarks…
Arcee's MergeKit: A Toolkit for Merging Large Language Models
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi +5
The rapid expansion of the open-source language model landscape presents an opportunity to merge the competencies of these model checkpoints by combining their parameters. Advances…