From the 2 of 64 linked papers with an AI index.
1 citations · 1 across the 25 of their papers we have counts for
4 papers · 1 filter
LLMQ: Efficient Lower-Precision Pretraining for Consumer GPUs
Erik Schultheis, Dan Alistarh
We present LLMQ, an end-to-end CUDA/C++ implementation for medium-sized language-model training, e.g. 3B to 32B parameters, on affordable, commodity GPUs. These devices are charact…
Breaking (Global) Barriers in Parallel Stochastic Optimization with Wait-Avoiding Group Averaging
Shigang Li, Tal Ben-Nun, Giorgi Nadiradze +4
Deep learning at scale is dominated by communication time. Distributing samples across nodes usually yields the best performance, but poses scaling challenges due to global informa…
Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operations
Shigang Li, Tal Ben-Nun, Salvatore Di Girolamo +2
Load imbalance pervasively exists in distributed deep learning training systems, either caused by the inherent imbalance in learned tasks or by the system itself. Traditional synch…
Game Dynamics and Equilibrium Computation in the Population Protocol Model
Dan Alistarh, Krishnendu Chatterjee, Mehrdad Karrabi +1
We initiate the study of game dynamics in the population protocol model: agents each maintain a current local strategy and interact in pairs uniformly at random. Upon each inte…