908 citations · 2.4k across the 149 of their papers we have counts for
8 papers · 1 filter
In Search of the Fastest Concurrent Union-Find Algorithm
Dan Alistarh, Alexander Fedorov, Nikita Koval
Union-Find (or Disjoint-Set Union) is one of the fundamental problems in computer science; it has been well-studied from both theoretical and practical perspectives in the sequenti…
Asynchronous Decentralized SGD with Quantized and Local Updates
Giorgi Nadiradze, Amirmojtaba Sabour, Peter Davies +2
Decentralized optimization is emerging as a viable alternative for scalable distributed machine learning, but also introduces new challenges in terms of synchronization costs. To t…
Powerset Convolutional Neural Networks
Chris Wendler, Dan Alistarh, Markus Püschel
We present a novel class of convolutional neural networks (CNNs) for set functions, i.e., data indexed with the powerset of a finite set. The convolutions are derived as linear, sh…
NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
Ali Ramezani-Kebrya, Fartash Faghri, Ilya Markov +3
As the size and complexity of models and datasets grow, so does the need for communication-efficient variants of stochastic gradient descent that can be deployed to perform paralle…
Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operations
Shigang Li, Tal Ben-Nun, Salvatore Di Girolamo +2
Load imbalance pervasively exists in distributed deep learning training systems, either caused by the inherent imbalance in learned tasks or by the system itself. Traditional synch…
Asynchronous Optimization Methods for Efficient Training of Deep Neural Networks with Guarantees
Vyacheslav Kungurtsev, Malcolm Egan, Bapi Chatterjee +1
Asynchronous distributed algorithms are a popular way to reduce synchronization costs in large-scale optimization, and in particular for neural network training. However, for nonsm…