89 citations · 194 across the 58 of their papers we have counts for
8 papers · 1 filter
FUSCO: High-Performance Distributed Data Shuffling via Transformation-Communication Fusion
Zhuoran Zhu, Chunyang Zhu, Hao Lin +9
Large-scale Mixture-of-Experts (MoE) models rely on \emph{expert parallelism} for efficient training and inference, which splits experts across devices and necessitates distributed…
Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reordering
Ke Hong, Xiuhong Li, Minxu Liu +9
Generative models have achieved remarkable success across various applications, driving the demand for multi-GPU computing. Inter-GPU communication becomes a bottleneck in multi-GP…
HETHUB: A Distributed Training System with Heterogeneous Cluster for Large-Scale Models
Si Xu, Zixiao Huang, Yan Zeng +11
Training large-scale models relies on a vast number of computing resources. For example, training the GPT-4 model (1.8 trillion parameters) requires 25000 A100 GPUs . It is a chall…
TorchSparse++: Efficient Training and Inference Framework for Sparse Convolution on GPUs
Haotian Tang, Shang Yang, Zhijian Liu +6
Sparse convolution plays a pivotal role in emerging workloads, including point cloud processing in AR/VR, autonomous driving, and graph understanding in recommendation systems. Sin…
Sgap: Towards Efficient Sparse Tensor Algebra Compilation for GPU
Genghan Zhang, Yuetong Zhao, Yanting Tao +6
Sparse compiler is a promising solution for sparse tensor algebra optimization. In compiler implementation, reduction in sparse-dense hybrid algebra plays a key role in performance…
Efficient Sparse Matrix Kernels based on Adaptive Workload-Balancing and Parallel-Reduction
Guyue Huang, Guohao Dai, Yu Wang +2
Sparse matrix-vector and matrix-matrix multiplication (SpMV and SpMM) are fundamental in both conventional (graph analytics, scientific computing) and emerging (sparse DNN, GNN) do…
GE-SpMM: General-purpose Sparse Matrix-Matrix Multiplication on GPUs for Graph Neural Networks
Guyue Huang, Guohao Dai, Yu Wang +1
Graph Neural Networks (GNNs) have achieved significant improvements in various domains. Sparse Matrix-Matrix multiplication (SpMM) is a fundamental operator in GNNs, which performs…
Enabling Efficient and Flexible FPGA Virtualization for Deep Learning in the Cloud
Shulin Zeng, Guohao Dai, Hanbo Sun +5
FPGAs have shown great potential in providing low-latency and energy-efficient solutions for deep neural network (DNN) inference applications. Currently, the majority of FPGA-based…