885 citations · 2.1k across the 60 of their papers we have counts for
5 papers · 1 filter
Finding Everything within Random Binary Networks
Kartik Sreenivasan, Shashank Rajput, Jy-yong Sohn +1
A recent work by Ramanujan et al. (2020) provides significant empirical evidence that sufficiently overparameterized, random neural networks contain untrained subnetworks that achi…
An Exponential Improvement on the Memorization Capacity of Deep Threshold Networks
Shashank Rajput, Kartik Sreenivasan, Dimitris Papailiopoulos +1
It is well known that modern deep neural networks are powerful enough to memorize datasets even when the labels have been randomized. Recently, Vershynin (2020) settled a long stan…
Pufferfish: Communication-efficient Models At No Extra Cost
Hongyi Wang, Saurabh Agarwal, Dimitris Papailiopoulos
To mitigate communication overheads in distributed model training, several studies propose the use of compressed stochastic gradients, usually achieved by sparsification or quantiz…
On the Utility of Gradient Compression in Distributed Training Systems
Saurabh Agarwal, Hongyi Wang, Shivaram Venkataraman +1
A rich body of prior work has highlighted the existence of communication bottlenecks in synchronous data-parallel training. To alleviate these bottlenecks, a long line of recent wo…
Permutation-Based SGD: Is Random Optimal?
Shashank Rajput, Kangwook Lee, Dimitris Papailiopoulos
A recent line of ground-breaking results for permutation-based SGD has corroborated a widely observed phenomenon: random permutations offer faster convergence than with-replacement…