5 papers · 1 filter
T-CCL: Resource Efficient and Performant Collective Communication using Tensor Memory Accelerator
Keyvan Dadashzadeh, Yuehong Zhou, Minyu Cui +1
Large transformer-based models increasingly depend on multi-GPU execution, which requires frequent collective communication among GPUs. Existing communication libraries often rely…
Analytical Resource Management for Fine-grained MoE Computation-Communication Overlap
Hongyu Liu, Minyu Cui, Miquel Pericas
Fine-grained computation--communication overlap in distributed Mixture-of-Experts (MoE) inference allows communication to begin as partial compute results become ready. However, co…
FlashAttention for Scalable Vector Architectures
Sonia Rani Gupta, Nikela Papadopoulou, Miquel Pericàs
Inference with transformer models on CPUs is increasingly important, especially for Small Language Models (SLMs), where vector architectures are emerging as a promising execution s…
DEFT: Joint Task Placement and DVFS for Energy-Efficient Multi-GPU Runtimes
Jing Chen, Miquel Pericas
Energy efficiency has become a first-order concern in modern high-performance computing systems, as it directly determines achievable throughput under fixed power budgets. Although…
Resource-aware Computation-Communication Overlap for multi-GPU ML Workloads
Minyu Cui, Miquel Pericas
The rapid growth of large-scale machine learning (ML) has made distributed training across multiple GPUs a fundamental component of modern ML systems. As model sizes and computatio…