299 citations · 354 across the 2 of their papers we have counts for
4 papers
Reducing Activation Recomputation in Large Transformer Models
Vijay Korthikanti, Jared Casper, Sangkug Lym +4
Training large transformer models is one of the most important computational challenges of modern AI. In this paper, we show how to significantly accelerate training of large trans…
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
Shaden Smith, Mostofa Patwary, Brandon Norick +17
Pretrained general-purpose language models can achieve state-of-the-art accuracies in various natural language processing domains by adapting to downstream tasks via zero-shot, few…
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
Deepak Narayanan, Mohammad Shoeybi, Jared Casper +9
Large language models have led to state-of-the-art accuracies across a range of tasks. However, training these models efficiently is challenging for two reasons: a) GPU memory capa…
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Mohammad Shoeybi, Mostofa Patwary, Raul Puri +3
Recent work in language modeling demonstrates that training large transformer models advances the state of the art in Natural Language Processing applications. However, very large…