483 citations · 3k across the 177 of their papers we have counts for
73 papers · 1 filter
What Bloats Your Floats? Right-Sizing Numerical Precision for Scientific Computing
Pratyai Mazumder, Alexandru Calotoiu, Marcin Copik +3
While FP64 (binary64) remains the standard representation of real numbers in scientific computing, hardware trends increasingly favor low-precision formats optimized for AI workloa…
FP64 Is All You Want, INT8 Is All You Need, FP4/6/8 Is All You Have
Pratyai Mazumder, Alexandru Calotoiu, Torsten Hoefler
Ozaki scheme II emulates FP64 matrix products with INT8 ones through residues modulo pairwise coprime moduli, and variants for FP8 and FP4 have followed. We treat these schemes as…
ADELIA: Automatic Differentiation for Efficient Laplace Inference Approximations
Afif Boudaoud, Lisa Gaedke-Merzhäuser, Alexandros Nikolaos Ziogas +6
Spatio-temporal Bayesian inference drives environmental and health sciences using latent Gaussian models. Integrated Nested Laplace Approximations (INLA) enable inference for these…
Design in Tiles: Automating GEMM Deployment on Tile-Based Many-PE Accelerators
Aofeng Shen, Chi Zhang, Yakup Budanaz +4
Tile-based many-Processing Element (PE) accelerators can achieve competitive performance on General Matrix Multiplication (GEMM), but they are extremely hard to program, as their o…
PICO: Performance Insights for Collective Operations
Saverio Pasqualoni, Tommaso Bonato, Lorenzo Piarulli +3
Collective operations are cornerstones of both HPC applications and large-scale AI training and inference, yet benchmarking them in a systematic and reproducible way remains diffic…
Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnects
Daniele De Sensi, Lorenzo Pichetti, Flavio Vella +11
Multi-GPU nodes are increasingly common in the rapidly evolving landscape of exascale supercomputers. On these systems, GPUs on the same node are connected through dedicated networ…
Network-Offloaded Bandwidth-Optimal Broadcast and Allgather for Distributed AI
Mikhail Khalilov, Salvatore Di Girolamo, Marcin Chrapek +3
In the Fully Sharded Data Parallel (FSDP) training pipeline, collective operations can be interleaved to maximize the communication/computation overlap. In this scenario, outstandi…
LLAMP: Assessing Network Latency Tolerance of HPC Applications with Linear Programming
Siyuan Shen, Langwen Huang, Marcin Chrapek +5
The shift towards high-bandwidth networks driven by AI workloads in data centers and HPC clusters has unintentionally aggravated network latency, adversely affecting the performanc…
Near-Optimal Wafer-Scale Reduce
Piotr Luczynski, Lukas Gianinazzi, Patrick Iff +3
Efficient Reduce and AllReduce communication collectives are a critical cornerstone of high-performance computing (HPC) applications. We present the first systematic investigation…
Low-Depth Spatial Tree Algorithms
Yves Baumann, Tal Ben-Nun, Maciej Besta +3
Contemporary accelerator designs exhibit a high degree of spatial localization, wherein two-dimensional physical distance determines communication costs between processing elements…
Arrow Matrix Decomposition: A Novel Approach for Communication-Efficient Sparse Matrix Multiplication
Lukas Gianinazzi, Alexandros Nikolaos Ziogas, Langwen Huang +9
We propose a novel approach to iterated sparse matrix dense matrix multiplication, a fundamental computational kernel in scientific computing and graph neural network training. In…
XaaS: Acceleration as a Service to Enable Productive High-Performance Cloud Computing
Torsten Hoefler, Marcin Copik, Pete Beckman +8
HPC and Cloud have evolved independently, specializing their innovations into performance or productivity. Acceleration as a Service (XaaS) is a recipe to empower both fields with…
Software Resource Disaggregation for HPC with Serverless Computing
Marcin Copik, Marcin Chrapek, Larissa Schmid +2
Aggregated HPC resources have rigid allocation systems and programming models which struggle to adapt to diverse and changing workloads. Consequently, HPC systems fail to efficient…
Cppless: Single-Source and High-Performance Serverless Programming in C++
Marcin Copik, Lukas Möller, Alexandru Calotoiu +1
The rise of serverless computing introduced a new class of scalable, elastic and widely available parallel workers in the cloud. Many systems and applications benefit from offloadi…
Swing: Short-cutting Rings for Higher Bandwidth Allreduce
Daniele De Sensi, Tommaso Bonato, David Saam +1
The allreduce collective operation accounts for a significant fraction of the runtime of workloads running on distributed systems. One factor determining its performance is the dis…
VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Cores
Roberto L. Castro, Andrei Ivanov, Diego Andrade +3
The increasing success and scaling of Deep Learning models demands higher computational efficiency and power. Sparsification can lead to both smaller models as well as higher compu…
Canary: Congestion-Aware In-Network Allreduce Using Dynamic Trees
Daniele De Sensi, Edgar Costa Molero, Salvatore Di Girolamo +2
The allreduce operation is an essential building block for many distributed applications, ranging from the training of deep learning models to scientific computing. In an allreduce…
Streaming Task Graph Scheduling for Dataflow Architectures
Tiziano De Matteis, Lukas Gianinazzi, Johannes de Fine Licht +1
Dataflow devices represent an avenue towards saving the control and data movement overhead of Load-Store Architectures. Various dataflow accelerators have been proposed, but how to…
FMI: Fast and Cheap Message Passing for Serverless Functions
Marcin Copik, Roman Böhringer, Alexandru Calotoiu +1
Serverless functions provide elastic scaling and a fine-grained billing model, making Function-as-a-Service (FaaS) an attractive programming model. However, for distributed jobs th…
A Theory of I/O-Efficient Sparse Neural Network Inference
Niels Gleinig, Tal Ben-Nun, Torsten Hoefler
As the accuracy of machine learning models increases at a fast rate, so does their demand for energy and compute resources. On a low level, the major part of these resources is con…
AutoDDL: Automatic Distributed Deep Learning with Near-Optimal Bandwidth Cost
Jinfan Chen, Shigang Li, Ran Gun +2
Recent advances in deep learning are driven by the growing scale of computation, data, and models. However, efficiently training large-scale models on distributed systems requires…
Myths and Legends in High-Performance Computing
Satoshi Matsuoka, Jens Domke, Mohamed Wahib +2
In this thought-provoking article, we discuss certain myths and legends that are folklore among members of the high-performance computing community. We gathered these myths from co…
Python FPGA Programming with Data-Centric Multi-Level Design
Johannes de Fine Licht, Tiziano De Matteis, Tal Ben-Nun +5
Although high-level synthesis (HLS) tools have significantly improved programmer productivity over hardware description languages, developing for FPGAs remains tedious and error pr…
Noise in the Clouds: Influence of Network Performance Variability on Application Scalability
Daniele De Sensi, Tiziano De Matteis, Konstantin Taranov +3
Cloud computing represents an appealing opportunity for cost-effective deployment of HPC workloads on the best-fitting hardware. However, although cloud and on-premise HPC systems…