Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models

Gil Avraham, Violetta Shevchenko, Hadi Mohaghegh Dolatabadi +9

Training large language models at the multi-billion to trillion parameter scale is confined to datacenters, where data-parallel (DP) and model-parallel (MP) techniques presume homo…