7 citations · 7 across the 7 of their papers we have counts for
4 papers · 1 filter
Latent Space Communication via K-V Cache Alignment
Lucio M. Dery, Zohar Yahav, Henry Prior +3
Solving increasingly complex problems with large language models (LLMs) necessitates a move beyond individual models and towards multi-model systems that can effectively collaborat…
DiPaCo: Distributed Path Composition
Arthur Douillard, Qixuan Feng, Andrei A. Rusu +7
Progress in machine learning (ML) has been fueled by scaling neural network models. This scaling has been enabled by ever more heroic feats of engineering, necessary for accommodat…
Asynchronous Local-SGD Training for Language Modeling
Bo Liu, Rachita Chhaparia, Arthur Douillard +5
Local stochastic gradient descent (Local-SGD), also referred to as federated averaging, is an approach to distributed optimization where each device performs more than one SGD upda…
DiLoCo: Distributed Low-Communication Training of Language Models
Arthur Douillard, Qixuan Feng, Andrei A. Rusu +6
Large language models (LLM) have become a critical component in many applications of machine learning. However, standard approaches to training LLM require a large number of tightl…