most citedMixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths

2 citations · 4 across the 7 of their papers we have counts for

collaborators
Showing cs.DCShow all

2 papers · 1 filter

cs.DC2025★ 1 cited

Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reordering

Ke Hong, Xiuhong Li, Minxu Liu +9

Generative models have achieved remarkable success across various applications, driving the demand for multi-GPU computing. Inter-GPU communication becomes a bottleneck in multi-GP…

cs.DC2024

HETHUB: A Distributed Training System with Heterogeneous Cluster for Large-Scale Models

Si Xu, Zixiao Huang, Yan Zeng +11

Training large-scale models relies on a vast number of computing resources. For example, training the GPT-4 model (1.8 trillion parameters) requires 25000 A100 GPUs . It is a chall…