483 citations · 3k across the 177 of their papers we have counts for
33 papers · 1 filter
How to Prune Your Language Model: Recovering Accuracy on the "Sparsity May Cry'' Benchmark
Eldar Kurtic, Torsten Hoefler, Dan Alistarh
Pruning large language models (LLMs) from the BERT family has emerged as a standard compression benchmark, and several pruning methods have been proposed for this task. The recent…
QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models
Saleh Ashkboos, Ilia Markov, Elias Frantar +5
Large Language Models (LLMs) from the GPT family have become extremely popular, leading to a race towards reducing their inference costs to allow for efficient local computation. Y…
HOT: Higher-Order Dynamic Graph Representation Learning with Efficient Transformers
Maciej Besta, Afonso Claudino Catarino, Lukas Gianinazzi +4
Many graph representation learning (GRL) problems are dynamic, with millions of edges added or removed per second. A fundamental workload in this setting is dynamic link prediction…
RapidChiplet: A Toolchain for Rapid Design Space Exploration of Chiplet Architectures
Patrick Iff, Benigna Bruggmann, Blaise Morel +3
Chiplet architectures are on the rise as they promise to overcome the scaling challenges of monolithic chips. A key component of such architectures is an efficient inter-chiplet in…
VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Cores
Roberto L. Castro, Andrei Ivanov, Diego Andrade +3
The increasing success and scaling of Deep Learning models demands higher computational efficiency and power. Sparsification can lead to both smaller models as well as higher compu…
Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models
Wenqi Jiang, Marco Zeller, Roger Waleffe +2
A Retrieval-Augmented Language Model (RALM) combines a large language model (LLM) with a vector database to retrieve context-specific knowledge during text generation. This strateg…
A High-Performance Design, Implementation, Deployment, and Evaluation of The Slim Fly Network
Nils Blach, Maciej Besta, Daniele De Sensi +10
Novel low-diameter network topologies such as Slim Fly (SF) offer significant cost and power advantages over the established Fat Tree, Clos, or Dragonfly. To spearhead the adoption…
Canary: Congestion-Aware In-Network Allreduce Using Dynamic Trees
Daniele De Sensi, Edgar Costa Molero, Salvatore Di Girolamo +2
The allreduce operation is an essential building block for many distributed applications, ranging from the training of deep learning models to scientific computing. In an allreduce…
Earth Virtualization Engines -- A Technical Perspective
Torsten Hoefler, Bjorn Stevens, Andreas F. Prein +17
Participants of the Berlin Summit on Earth Virtualization Engines (EVEs) discussed ideas and concepts to improve our ability to cope with climate change. EVEs aim to provide intera…
OSMOSIS: Enabling Multi-Tenancy in Datacenter SmartNICs
Mikhail Khalilov, Marcin Chrapek, Siyuan Shen +7
Multi-tenancy is essential for unleashing SmartNIC's potential in datacenters. Our systematic analysis in this work shows that existing on-path SmartNICs have resource multiplexing…
Cached Operator Reordering: A Unified View for Fast GNN Training
Julia Bazinska, Andrei Ivanov, Tal Ben-Nun +4
Graph Neural Networks (GNNs) are a powerful tool for handling structured graph data and addressing tasks such as node classification, graph classification, and clustering. However,…
Graph of Thoughts: Solving Elaborate Problems with Large Language Models
Maciej Besta, Nils Blach, Ales Kubicek +8
We introduce Graph of Thoughts (GoT): a framework that advances prompting capabilities in large language models (LLMs) beyond those offered by paradigms such as Chain-of-Thought or…
Differentiable Transportation Pruning
Yunqiang Li, Jan C. van Gemert, Torsten Hoefler +3
Deep learning algorithms are increasingly employed at the edge. However, edge devices are resource constrained and thus require efficient deployment of deep neural networks. Prunin…
Maximum Flows in Parametric Graph Templates
Tal Ben-Nun, Lukas Gianinazzi, Torsten Hoefler +1
Execution graphs of parallel loop programs exhibit a nested, repeating structure. We show how such graphs that are the result of nested repetition can be represented by succinct pa…
Co-design Hardware and Algorithm for Vector Search
Wenqi Jiang, Shigang Li, Yu Zhu +8
Vector search has emerged as the foundation for large-scale information retrieval and machine learning systems, with search engines like Google and Bing processing tens of thousand…
Disentangling Hype from Practicality: On Realistically Achieving Quantum Advantage
Torsten Hoefler, Thomas Haener, Matthias Troyer
Quantum computers offer a new paradigm of computing with the potential to vastly outperform any imagineable classical computer. This has caused a gold rush towards new quantum algo…
FuzzyFlow: Leveraging Dataflow To Find and Squash Program Optimization Bugs
Philipp Schaad, Timo Schneider, Tal Ben-Nun +3
The current hardware landscape and application scale is driving performance engineers towards writing bespoke optimizations. Verifying such optimizations, and generating minimal fa…
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian +6
Recent advances in large language model (LLM) pretraining have led to high-quality LLMs with impressive abilities. By compressing such LLMs via quantization to 3-4 bits per paramet…
Streaming Task Graph Scheduling for Dataflow Architectures
Tiziano De Matteis, Lukas Gianinazzi, Johannes de Fine Licht +1
Dataflow devices represent an avenue towards saving the control and data movement overhead of Load-Store Architectures. Various dataflow accelerators have been proposed, but how to…
Bridging Control-Centric and Data-Centric Optimization
Tal Ben-Nun, Berke Ates, Alexandru Calotoiu +1
With the rise of specialized hardware and new programming languages, code optimization has shifted its focus towards promoting data locality. Most production-grade compilers adopt…
FMI: Fast and Cheap Message Passing for Serverless Functions
Marcin Copik, Roman Böhringer, Alexandru Calotoiu +1
Serverless functions provide elastic scaling and a fine-grained billing model, making Function-as-a-Service (FaaS) an attractive programming model. However, for distributed jobs th…
ASDL: A Unified Interface for Gradient Preconditioning in PyTorch
Kazuki Osawa, Satoki Ishikawa, Rio Yokota +2
Gradient preconditioning is a key technique to integrate the second-order information into gradients for improving and extending gradient-based learning algorithms. In deep learnin…
The Graph Database Interface: Scaling Online Transactional and Analytical Graph Workloads to Hundreds of Thousands of Cores
Maciej Besta, Robert Gerstenberger, Marc Fischer +9
Graph databases (GDBs) are crucial in academic and industry applications. The key challenges in developing GDBs are achieving high performance, scalability, programmability, and po…
Sparse Stream Semantic Registers: A Lightweight ISA Extension Accelerating General Sparse Linear Algebra
Paul Scheffler, Florian Zaruba, Fabian Schuiki +2
Sparse linear algebra is crucial in many application domains, but challenging to handle efficiently in both software and hardware, with one- and two-sided operand sparsity handled…