Showing cs.DCShow all
3 papers · 1 filter
cs.DC2026
Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism
Sajal Dash, Feiyi Wang
Frontier models increasingly adopt Mixture-of-Experts (MoE) architectures to achieve large-model performance at reduced cost. However, training MoE models on HPC platforms is hinde…
cs.DC2023
Optimizing Distributed Training on Frontier for Large Language Models
Sajal Dash, Isaac Lyngaas, Junqi Yin +5
Large language models (LLMs) have demonstrated remarkable success as foundational models, benefiting various downstream applications through fine-tuning. Recent studies on loss sca…
cs.DC2023★ 3 cited
Ultra-Long Sequence Distributed Transformer
Xiao Wang, Isaac Lyngaas, Aristeidis Tsaris +7
Transformer models trained on long sequences often achieve higher accuracy than short sequences. Unfortunately, conventional transformers struggle with long sequence training due t…