7 citations · 13 across the 7 of their papers we have counts for
5 papers · 1 filter
MoE-Hub: Taming Software Complexity for Seamless MoE Overlap with Hardware-Accelerated Communication on Multi-GPU Systems
Zhuoshan Zhou, Chen Zhang, Shuyi Zhang +10
The Mixture-of-Experts (MoE) architecture is crucial for scaling large language models, but its scalability is severely limited by inter-GPU communication bottlenecks in multi-GPU…
Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUs
Qijun Zhang, Chen Zhang, Zhuoshan Zhou +10
Mixture-of-Experts (MoE) has been adopted by many leading large models to reduce computational requirements. However, frequent inter-GPU communication in MoE expert parallelism (EP…
UB-Mesh: a Hierarchically Localized nD-FullMesh Datacenter Network Architecture
Heng Liao, Bingyang Liu, Xianping Chen +31
As the Large-scale Language Models (LLMs) continue to scale, the requisite computational power and bandwidth escalate. To address this, we introduce UB-Mesh, a novel AI datacenter…
Energon: Towards Efficient Acceleration of Transformers Using Dynamic Sparse Attention
Zhe Zhou, Junlin Liu, Zhenyu Gu +1
In recent years, transformer models have revolutionized Natural Language Processing (NLP) and shown promising performance on Computer Vision (CV) tasks. Despite their effectiveness…
METRO: A Software-Hardware Co-Design of Interconnections for Spatial DNN Accelerators
Zhao Wang, Jingchen Zhu, Zhe Zhou +1
Tiled spatial architectures have proved to be an effective solution to build large-scale DNN accelerators. In particular, interconnections between tiles are critical for high perfo…