151 citations · 389 across the 13 of their papers we have counts for
20 papers · 1 filter
Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam
Yucheng Lu, Conglong Li, Minjia Zhang +2
1-bit gradient compression and local steps are two representative techniques that enable drastic communication reduction in distributed SGD. Their benefits, however, remain an open…
Variance Reduced Training with Stratified Sampling for Forecasting Models
Yucheng Lu, Youngsuk Park, Lifan Chen +3
In large-scale time series forecasting, one often encounters the situation where the temporal patterns of time series, while drifting over time, differ from one another in the same…
Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
Johan Bjorck, Xiangyu Chen, Christopher De Sa +2
Low-precision training has become a popular approach to reduce compute requirements, memory footprint, and energy consumption in supervised learning. In contrast, this promising ap…
Revisiting BFloat16 Training
Pedram Zamirai, Jian Zhang, Christopher R. Aberger +1
State-of-the-art generic low-precision training algorithms use a mix of 16-bit and 32-bit precision, creating the folklore that 16-bit hardware compute units alone are not enough t…
MixML: A Unified Analysis of Weakly Consistent Parallel Learning
Yucheng Lu, Jack Nash, Christopher De Sa
Parallelism is a ubiquitous method for accelerating machine learning algorithms. However, theoretical analysis of parallel learning is usually done in an algorithm- and protocol-sp…
Moniqua: Modulo Quantized Communication in Decentralized SGD
Yucheng Lu, Christopher De Sa
Running Stochastic Gradient Descent (SGD) in a decentralized fashion has shown promising results. In this paper we propose Moniqua, a technique that allows decentralized SGD to use…