78 citations · 300 across the 10 of their papers we have counts for
3 papers · 1 filter
PipeMare: Asynchronous Pipeline Parallel DNN Training
Bowen Yang, Jian Zhang, Jonathan Li +3
Pipeline parallelism (PP) when training neural networks enables larger models to be partitioned spatially, leading to both lower network communication and overall higher hardware u…
On the Downstream Performance of Compressed Word Embeddings
Avner May, Jian Zhang, Tri Dao +1
Compressing word embeddings is important for deploying NLP models in memory-constrained settings. However, understanding what makes compressed embeddings perform well on downstream…
Low-Memory Neural Network Training: A Technical Report
Nimit S. Sohoni, Christopher R. Aberger, Megan Leszczynski +2
Memory is increasingly often the bottleneck when training neural network models. Despite this, techniques to lower the overall memory requirements of training have been less widely…