259 citations · 1.2k across the 48 of their papers we have counts for
6 papers · 1 filter
DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing
Conglong Li, Zhewei Yao, Xiaoxia Wu +4
Recent advances on deep learning models come at the price of formidable training cost. The increasing model size is one of the root causes, but another less-emphasized fact is that…
Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformers
Zhewei Yao, Xiaoxia Wu, Conglong Li +4
Large-scale transformer models have become the de-facto architectures for various machine learning applications, e.g., CV and NLP. However, those large models also introduce prohib…
BiFeat: Supercharge GNN Training via Graph Feature Quantization
Yuxin Ma, Ping Gong, Jun Yi +4
Graph Neural Networks (GNNs) is a promising approach for applications with nonEuclidean data. However, training GNNs on large scale graphs with hundreds of millions nodes is both r…
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang +3
How to efficiently serve ever-larger trained natural language models in practice has become exceptionally challenging even for powerful cloud servers due to their prohibitive memor…
Extreme Compression for Pre-trained Transformers Made Simple and Efficient
Xiaoxia Wu, Zhewei Yao, Minjia Zhang +2
Extreme compression, particularly ultra-low bit precision (binary/ternary) quantization, has been proposed to fit large NLP models on resource-constraint devices. However, to prese…
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
Samyam Rajbhandari, Conglong Li, Zhewei Yao +5
As the training of giant dense models hits the boundary on the availability and capability of the hardware resources today, Mixture-of-Experts (MoE) models become one of the most p…