341 citations · 1.1k across the 43 of their papers we have counts for
46 papers
Arrow Matrix Decomposition: A Novel Approach for Communication-Efficient Sparse Matrix Multiplication
Lukas Gianinazzi, Alexandros Nikolaos Ziogas, Langwen Huang +9
We propose a novel approach to iterated sparse matrix dense matrix multiplication, a fundamental computational kernel in scientific computing and graph neural network training. In…
VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Cores
Roberto L. Castro, Andrei Ivanov, Diego Andrade +3
The increasing success and scaling of Deep Learning models demands higher computational efficiency and power. Sparsification can lead to both smaller models as well as higher compu…
ComPile: A Large IR Dataset from Production Sources
Aiden Grossman, Ludger Paehler, Konstantinos Parasyris +6
Code is increasingly becoming a core data modality of modern machine learning research impacting not only the way we write code with conversational agents like OpenAI's ChatGPT, Go…
Cached Operator Reordering: A Unified View for Fast GNN Training
Julia Bazinska, Andrei Ivanov, Tal Ben-Nun +4
Graph Neural Networks (GNNs) are a powerful tool for handling structured graph data and addressing tasks such as node classification, graph classification, and clustering. However,…
Maximum Flows in Parametric Graph Templates
Tal Ben-Nun, Lukas Gianinazzi, Torsten Hoefler +1
Execution graphs of parallel loop programs exhibit a nested, repeating structure. We show how such graphs that are the result of nested repetition can be represented by succinct pa…
FuzzyFlow: Leveraging Dataflow To Find and Squash Program Optimization Bugs
Philipp Schaad, Timo Schneider, Tal Ben-Nun +3
The current hardware landscape and application scale is driving performance engineers towards writing bespoke optimizations. Verifying such optimizations, and generating minimal fa…
Bridging Control-Centric and Data-Centric Optimization
Tal Ben-Nun, Berke Ates, Alexandru Calotoiu +1
With the rise of specialized hardware and new programming languages, code optimization has shifted its focus towards promoting data locality. Most production-grade compilers adopt…
STen: Productive and Efficient Sparsity in PyTorch
Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun +2
As deep learning models grow, sparsity is becoming an increasingly critical component of deep neural networks, enabling improved performance and reduced storage. However, existing…
Performance Embeddings: A Similarity-based Approach to Automatic Performance Optimization
Lukas Trümper, Tal Ben-Nun, Philipp Schaad +2
Performance optimization is an increasingly challenging but often repetitive task. While each platform has its quirks, the underlying code transformations rely on data movement and…
A Theory of I/O-Efficient Sparse Neural Network Inference
Niels Gleinig, Tal Ben-Nun, Torsten Hoefler
As the accuracy of machine learning models increases at a fast rate, so does their demand for energy and compute resources. On a low level, the major part of these resources is con…
Python FPGA Programming with Data-Centric Multi-Level Design
Johannes de Fine Licht, Tiziano De Matteis, Tal Ben-Nun +5
Although high-level synthesis (HLS) tools have significantly improved programmer productivity over hardware description languages, developing for FPGAs remains tedious and error pr…
Temporal Vectorization: A Compiler Approach to Automatic Multi-Pumping
Carl-Johannes Johnsen, Tiziano De Matteis, Tal Ben-Nun +2
The multi-pumping resource sharing technique can overcome the limitations commonly found in single-clocked FPGA designs by allowing hardware components to operate at a higher clock…
Boosting Performance Optimization with Interactive Data Movement Visualization
Philipp Schaad, Tal Ben-Nun, Torsten Hoefler
Optimizing application performance in today's hardware architecture landscape is an important, but increasingly complex task, often requiring detailed performance analyses. In part…
Deinsum: Practically I/O Optimal Multilinear Algebra
Alexandros Nikolaos Ziogas, Grzegorz Kwasniewski, Tal Ben-Nun +2
Multilinear algebra kernel performance on modern massively-parallel systems is determined mainly by data movement. However, deriving data movement-optimal distributed schedules for…
ENS-10: A Dataset For Post-Processing Ensemble Weather Forecasts
Saleh Ashkboos, Langwen Huang, Nikoli Dryden +5
Post-processing ensemble prediction systems can improve the reliability of weather forecasting, especially for extreme event prediction. In recent years, different machine learning…
The spatial computer: A model for energy-efficient parallel computation
Lukas Gianinazzi, Tal Ben-Nun, Maciej Besta +4
We present a new parallel model of computation suitable for spatial architectures, for which the energy used for communication heavily depends on the distance of the communicating…
Productive Performance Engineering for Weather and Climate Modeling with Python
Tal Ben-Nun, Linus Groner, Florian Deconinck +11
Earth system models are developed with a tight coupling to target hardware, often containing specialized code predicated on processor characteristics. This coupling stems from usin…
Lifting C Semantics for Dataflow Optimization
Alexandru Calotoiu, Tal Ben-Nun, Grzegorz Kwasniewski +4
C is the lingua franca of programming and almost any device can be programmed using C. However, programming mod-ern heterogeneous architectures such as multi-core CPUs and GPUs req…
A Data-Centric Optimization Framework for Machine Learning
Oliver Rausch, Tal Ben-Nun, Nikoli Dryden +3
Rapid progress in deep learning is leading to a diverse set of quickly changing models, with a dramatically growing demand for compute. However, as frameworks specialize performanc…
On the Parallel I/O Optimality of Linear Algebra Kernels: Near-Optimal Matrix Factorizations
Grzegorz Kwasniewski, Marko Kabić, Tal Ben-Nun +8
Matrix factorizations are among the most important building blocks of scientific computing. State-of-the-art libraries, however, are not communication-optimal, underutilizing curre…
Learning Combinatorial Node Labeling Algorithms
Lukas Gianinazzi, Maximilian Fries, Nikoli Dryden +3
We present a novel neural architecture to solve graph optimization problems where the solution consists of arbitrary node labels, allowing us to solve hard problems like graph colo…
Pebbles, Graphs, and a Pinch of Combinatorics: Towards Tight I/O Lower Bounds for Statically Analyzable Programs
Grzegorz Kwasniewski, Tal Ben-Nun, Lukas Gianinazzi +5
Determining I/O lower bounds is a crucial step in obtaining communication-efficient parallel algorithms, both across the memory hierarchy and between processors. Current approaches…
Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun +2
The growing energy and performance costs of deep learning have driven the community to reduce the size of neural networks by selectively pruning components. Similarly to their biol…
Clairvoyant Prefetching for Distributed Machine Learning I/O
Nikoli Dryden, Roman Böhringer, Tal Ben-Nun +1
I/O is emerging as a major bottleneck for machine learning training, especially in distributed environments. Indeed, at large scale, I/O takes as much as 85% of training time. Addr…