341 citations · 1.1k across the 43 of their papers we have counts for
14 papers · 1 filter
Cached Operator Reordering: A Unified View for Fast GNN Training
Julia Bazinska, Andrei Ivanov, Tal Ben-Nun +4
Graph Neural Networks (GNNs) are a powerful tool for handling structured graph data and addressing tasks such as node classification, graph classification, and clustering. However,…
STen: Productive and Efficient Sparsity in PyTorch
Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun +2
As deep learning models grow, sparsity is becoming an increasingly critical component of deep neural networks, enabling improved performance and reduced storage. However, existing…
ENS-10: A Dataset For Post-Processing Ensemble Weather Forecasts
Saleh Ashkboos, Langwen Huang, Nikoli Dryden +5
Post-processing ensemble prediction systems can improve the reliability of weather forecasting, especially for extreme event prediction. In recent years, different machine learning…
A Data-Centric Optimization Framework for Machine Learning
Oliver Rausch, Tal Ben-Nun, Nikoli Dryden +3
Rapid progress in deep learning is leading to a diverse set of quickly changing models, with a dramatically growing demand for compute. However, as frameworks specialize performanc…
Learning Combinatorial Node Labeling Algorithms
Lukas Gianinazzi, Maximilian Fries, Nikoli Dryden +3
We present a novel neural architecture to solve graph optimization problems where the solution consists of arbitrary node labels, allowing us to solve hard problems like graph colo…
Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun +2
The growing energy and performance costs of deep learning have driven the community to reduce the size of neural networks by selectively pruning components. Similarly to their biol…
Data Movement Is All You Need: A Case Study on Optimizing Transformers
Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun +2
Transformers are one of the most important machine learning workloads today. Training one is a very compute-intensive task, often taking days or weeks, and significant attention ha…
Deep Learning for Post-Processing Ensemble Weather Forecasts
Peter Grönquist, Chengyuan Yao, Tal Ben-Nun +4
Quantifying uncertainty in weather forecasts is critical, especially for predicting extreme weather events. This is typically accomplished with ensemble prediction systems, which c…
ProGraML: Graph-based Deep Learning for Program Optimization and Analysis
Chris Cummins, Zacharias V. Fisches, Tal Ben-Nun +2
The increasing complexity of computing systems places a tremendous burden on optimizing compilers, requiring ever more accurate and aggressive optimizations. Machine learning offer…
Predicting Weather Uncertainty with Deep Convnets
Peter Grönquist, Tal Ben-Nun, Nikoli Dryden +4
Modern weather forecast models perform uncertainty quantification using ensemble prediction systems, which collect nonparametric statistics based on multiple perturbed simulations.…
Augment your batch: better training with larger batches
Elad Hoffer, Tal Ben-Nun, Itay Hubara +3
Large-batch SGD is important for scaling training of deep neural networks. However, without fine-tuning hyperparameter schedules, the generalization of the model may be hampered. W…
Neural Code Comprehension: A Learnable Representation of Code Semantics
Tal Ben-Nun, Alice Shoshana Jakobovits, Torsten Hoefler
With the recent success of embeddings in natural language processing, research has been conducted into applying similar methods to code analysis. Most works attempt to process the…
μ-cuDNN: Accelerating Deep Learning Frameworks with Micro-Batching
Yosuke Oyama, Tal Ben-Nun, Torsten Hoefler +1
NVIDIA cuDNN is a low-level library that provides GPU kernels frequently used in deep learning. Specifically, cuDNN implements several equivalent convolution algorithms, whose perf…
Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
Tal Ben-Nun, Torsten Hoefler
Deep Neural Networks (DNNs) are becoming an important tool in modern computing applications. Accelerating their training is a major challenge and techniques range from distributed…