131 citations · 346 across the 18 of their papers we have counts for
19 papers
Efficient Training of Large Language Models on Distributed Infrastructures: A Survey
Jiangfei Duan, Shuo Zhang, Zerui Wang +13
Large Language Models (LLMs) like GPT and LLaMA are revolutionizing the AI industry with their sophisticated capabilities. Training these models requires vast GPU clusters and sign…
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
Pan Zhang, Xiaoyi Dong, Yuhang Zang +24
We present InternLM-XComposer-2.5 (IXC-2.5), a versatile large-vision language model that supports long-contextual input and output. IXC-2.5 excels in various text-image comprehens…
InternLM2 Technical Report
Zheng Cai, Maosong Cao, Haojiong Chen +97
The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has sparked discussions on the advent of Artificial General Intelligence (AGI). However, replicating such advan…
InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding
Qiaoling Chen, Diandian Gu, Guoteng Wang +8
Large language models (LLMs) with long sequences begin to power more and more fundamentally new applications we use every day. Existing methods for long-sequence LLM training are n…
AMSP: Reducing Communication Overhead of ZeRO for Efficient LLM Training
Qiaoling Chen, Qinghao Hu, Guoteng Wang +8
Training large language models (LLMs) encounters challenges in GPU memory consumption due to the high memory requirements of model states. The widely used Zero Redundancy Optimizer…
Boosting Distributed Full-graph GNN Training with Asynchronous One-bit Communication
Meng Zhang, Qinghao Hu, Peng Sun +2
Training Graph Neural Networks (GNNs) on large graphs is challenging due to the conflict between the high memory demand and limited GPU memory. Recently, distributed full-graph GNN…
Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision
Wei Gao, Qinghao Hu, Zhisheng Ye +5
Deep learning (DL) shows its prosperity in a wide variety of fields. The development of a DL model is a time-consuming and resource-intensive procedure. Hence, dedicated GPU accele…
A Simulation Platform for Multi-tenant Machine Learning Services on Thousands of GPUs
Ruofan Liang, Bingsheng He, Shengen Yan +1
Multi-tenant machine learning services have become emerging data-intensive workloads in data centers with heavy usage of GPU resources. Due to the large scale, many tuning paramete…
Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenters
Qinghao Hu, Peng Sun, Shengen Yan +2
Modern GPU datacenters are critical for delivering Deep Learning (DL) models and services in both the research community and industry. When operating a datacenter, optimization of…
ModelCI-e: Enabling Continual Learning in Deep Learning Serving Systems
Yizheng Huang, Huaizheng Zhang, Yonggang Wen +2
MLOps is about taking experimental ML models to production, i.e., serving the models to actual users. Unfortunately, existing ML serving systems do not adequately handle the dynami…
Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes
Peng Sun, Wansen Feng, Ruobing Han +2
It is important to scale out deep neural network (DNN) training for reducing model training time. The high communication overhead is one of the major performance bottlenecks for di…
GraphMP: I/O-Efficient Big Graph Analytics on a Single Commodity Machine
Peng Sun, Yonggang Wen, Ta Nguyen Binh Duong +1
Recent studies showed that single-machine graph processing systems can be as highly competitive as cluster-based approaches on large-scale problems. While several out-of-core graph…
Speeding-up Age Estimation in Intelligent Demographics System via Network Optimization
Zhenzhen Hui, Peng Sun, Yonggang Wen
Age estimation is a difficult task which requires the automatic detection and interpretation of facial features. Recently, Convolutional Neural Networks (CNNs) have made remarkable…
On Distributed Algorithms for Cost-Efficient Data Center Placement in Cloud Computing
Wuqiong Luo, Wee Peng Tay, Peng Sun +1
The increasing popularity of cloud computing has resulted in a proliferation of data centers. Effective placement of data centers improves network performance and minimizes clients…
GraphMP: An Efficient Semi-External-Memory Big Graph Processing System on a Single Machine
Peng Sun, Yonggang Wen, Ta Nguyen Binh Duong +1
Recent studies showed that single-machine graph processing systems can be as highly competitive as cluster-based approaches on large-scale problems. While several out-of-core graph…
GraphH: High Performance Big Graph Analytics in Small Clusters
Peng Sun, Yonggang Wen, Ta Nguyen Binh Duong +1
It is common for real-world applications to analyze big graphs using distributed graph processing systems. Popular in-memory systems require an enormous amount of resources to hand…
Towards Distributed Machine Learning in Shared Clusters: A Dynamically-Partitioned Approach
Peng Sun, Yonggang Wen, Ta Nguyen Binh Duong +1
Many cluster management systems (CMSs) have been proposed to share a single cluster with multiple distributed computing systems. However, none of the existing approaches can handle…
A Chunk Caching Location and Searching Scheme in Content Centric Networking
Yang Li, Tao Lin, Hui Tang +1
Content Centric Networking (CCN) is a new network infrastructure around content dissemination and retrieval, shift from host addresses to named data. Each CCN router has a cache to…
MetaFlow: a Scalable Metadata Lookup Service for Distributed File Systems in Data Centers
Peng Sun, Yonggang Wen, Ta Nguyen Binh Duong +1
In large-scale distributed file systems, efficient meta- data operations are critical since most file operations have to interact with metadata servers first. In existing distribut…