7 citations · 14 across the 4 of their papers we have counts for
4 papers
DiPaCo: Distributed Path Composition
Arthur Douillard, Qixuan Feng, Andrei A. Rusu +7
Progress in machine learning (ML) has been fueled by scaling neural network models. This scaling has been enabled by ever more heroic feats of engineering, necessary for accommodat…
Asynchronous Local-SGD Training for Language Modeling
Bo Liu, Rachita Chhaparia, Arthur Douillard +5
Local stochastic gradient descent (Local-SGD), also referred to as federated averaging, is an approach to distributed optimization where each device performs more than one SGD upda…
DiLoCo: Distributed Low-Communication Training of Language Models
Arthur Douillard, Qixuan Feng, Andrei A. Rusu +6
Large language models (LLM) have become a critical component in many applications of machine learning. However, standard approaches to training LLM require a large number of tightl…
A data-driven approach for learning to control computers
Peter C Humphreys, David Raposo, Toby Pohlen +8
It would be useful for machines to use computers as humans do so that they can aid us in everyday tasks. This is a setting in which there is also the potential to leverage large-sc…