5 papers
BurstEngine: an Efficient Distributed Framework for Training Transformers on Extremely Long Sequences of over 1M Tokens
Ao Sun, Weilin Zhao, Xu Han +4
Existing methods for training LLMs on long-sequence data, such as Tensor Parallelism and Context Parallelism, exhibit low Model FLOPs Utilization as sequence lengths and number of…
FairSIN: Achieving Fairness in Graph Neural Networks through Sensitive Information Neutralization
Cheng Yang, Jixi Liu, Yunhe Yan +1
Despite the remarkable success of graph neural networks (GNNs) in modeling graph-structured data, like other machine learning models, GNNs are also susceptible to making biased pre…
Graph Foundation Models: Concepts, Opportunities and Challenges
Jiawei Liu, Cheng Yang, Zhiyuan Lu +8
Foundation models have emerged as critical components in a variety of artificial intelligence applications, and showcase significant success in natural language processing and seve…
Data-centric Graph Learning: A Survey
Yuxin Guo, Deyu Bo, Cheng Yang +5
The history of artificial intelligence (AI) has witnessed the significant impact of high-quality data on various deep learning models, such as ImageNet for AlexNet and ResNet. Rece…
Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training
Ao Sun, Weilin Zhao, Xu Han +5
The emergence of large language models (LLMs) relies heavily on distributed training strategies, among which pipeline parallelism plays a crucial role. As LLMs' training sequence l…