13 citations · 28 across the 10 of their papers we have counts for
10 papers · 1 filter
SHIRO: Near-Optimal Communication Strategies for Distributed Sparse Matrix Multiplication
Chen Zhuang, Lingqi Zhang, Benjamin Brock +5
Distributed Sparse Matrix-Matrix Multiplication (SpMM) is a fundamental operation in high-performance computing and deep learning applications. The major performance bottleneck in…
Can Tensor Cores Benefit Memory-Bound Kernels? (No!)
Lingqi Zhang, Jiajun Huang, Sheng Di +2
Tensor cores are specialized processing units within GPUs that have demonstrated significant efficiency gains in compute-bound applications such as Deep Learning Training by accele…
Scaling Large-scale GNN Training to Thousands of Processors on CPU-based Supercomputers
Chen Zhuang, Lingqi Zhang, Du Wu +8
Graph Convolutional Networks (GCNs), particularly for large-scale graphs, are crucial across numerous domains. However, training distributed full-batch GCNs on large-scale graphs s…
Exploiting Scratchpad Memory for Deep Temporal Blocking: A case study for 2D Jacobian 5-point iterative stencil kernel (j2d5pt)
Lingqi Zhang, Mohamed Wahib, Peng Chen +4
General Purpose Graphics Processing Units (GPGPU) are used in most of the top systems in HPC. The total capacity of scratchpad memory has increased by more than 40 times in the las…
Revisiting Temporal Blocking Stencil Optimizations
Lingqi Zhang, Mohamed Wahib, Peng Chen +4
Iterative stencils are used widely across the spectrum of High Performance Computing (HPC) applications. Many efforts have been put into optimizing stencil GPU kernels, given the p…
At the Locus of Performance: Quantifying the Effects of Copious 3D-Stacked Cache on HPC Workloads
Jens Domke, Emil Vatai, Balazs Gerofi +9
Over the last three decades, innovations in the memory subsystem were primarily targeted at overcoming the data movement bottleneck. In this paper, we focus on a specific market tr…
PERKS: a Locality-Optimized Execution Model for Iterative Memory-bound GPU Applications
Lingqi Zhang, Mohamed Wahib, Peng Chen +4
Iterative memory-bound solvers commonly occur in HPC codes. Typical GPU implementations have a loop on the host side that invokes the GPU kernel as much as time/algorithm steps the…
Matrix Engines for High Performance Computing:A Paragon of Performance or Grasping at Straws?
Jens Domke, Emil Vatai, Aleksandr Drozd +8
Matrix engines or units, in different forms and affinities, are becoming a reality in modern processors; CPUs and otherwise. The current and dominant algorithmic approach to Deep L…
Scaling Distributed Deep Learning Workloads beyond the Memory Capacity with KARMA
Mohamed Wahib, Haoyu Zhang, Truong Thao Nguyen +5
The dedicated memory of hardware accelerators can be insufficient to store all weights and/or intermediate states of large deep learning models. Although model parallelism is a via…
A Study of Single and Multi-device Synchronization Methods in Nvidia GPUs
Lingqi Zhang, Mohamed Wahib, Haoyu Zhang +1
GPUs are playing an increasingly important role in general-purpose computing. Many algorithms require synchronizations at different levels of granularity in a single GPU. Additiona…