8 citations · 12 across the 6 of their papers we have counts for
7 papers
Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism
Sajal Dash, Feiyi Wang
Frontier models increasingly adopt Mixture-of-Experts (MoE) architectures to achieve large-model performance at reduced cost. However, training MoE models on HPC platforms is hinde…
X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms
Yueming Yuan, Ahan Gupta, Jianping Li +3
Emerging expert-specialized Mixture-of-Experts (MoE) architectures, such as DeepSeek-MoE, deliver strong model quality through fine-grained expert segmentation and large top-k rout…
A Parallel Alternative for Energy-Efficient Neural Network Training and Inferencing
Sudip K. Seal, Maksudul Alam, Jorge Ramirez +2
Energy efficiency of training and inferencing with large neural network models is a critical challenge facing the future of sustainable large-scale machine learning workloads. This…
Scalable Artificial Intelligence for Science: Perspectives, Methods and Exemplars
Wesley Brewer, Aditya Kashi, Sajal Dash +4
In a post-ChatGPT world, this paper explores the potential of leveraging scalable artificial intelligence for scientific discovery. We propose that scaling up artificial intelligen…
Optimizing Distributed Training on Frontier for Large Language Models
Sajal Dash, Isaac Lyngaas, Junqi Yin +5
Large language models (LLMs) have demonstrated remarkable success as foundational models, benefiting various downstream applications through fine-tuning. Recent studies on loss sca…
Ultra-Long Sequence Distributed Transformer
Xiao Wang, Isaac Lyngaas, Aristeidis Tsaris +7
Transformer models trained on long sequences often achieve higher accuracy than short sequences. Unfortunately, conventional transformers struggle with long sequence training due t…
DeepSpeed4Science Initiative: Enabling Large-Scale Scientific Discovery through Sophisticated AI System Technologies
Shuaiwen Leon Song, Bonnie Kruft, Minjia Zhang +89
In the upcoming decade, deep learning may revolutionize the natural sciences, enhancing our capacity to model and predict natural occurrences. This could herald a new era of scient…