124 citations · 311 across the 19 of their papers we have counts for
19 papers
Tensor Algebra Processing Primitives (TAPP): Towards a Standard for Tensor Operations
Jan Brandejs, Niklas Hörnblad, Edward F. Valeev +4
To address the absence of a universal standard interface for tensor operations, we introduce the Tensor Algebra Processing Primitives (TAPP), a C-based interface designed to decoup…
Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures
Evangelos Georganas, Dhiraj Kalamkar, Kirill Voronin +5
During the past decade, Deep Learning (DL) algorithms, programming systems and hardware have converged with the High Performance Computing (HPC) counterparts. Nevertheless, the pro…
FP8 Formats for Deep Learning
Paulius Micikevicius, Dusan Stosic, Neil Burgess +12
FP8 is a natural progression for accelerating deep learning training inference beyond the 16-bit formats common in modern processors. In this paper we propose an 8-bit floating poi…
FPGA-based AI Smart NICs for Scalable Distributed AI Training Systems
Rui Ma, Evangelos Georganas, Alexander Heinecke +2
Rapid advances in artificial intelligence (AI) technology have led to significant accuracy improvements in a myriad of application domains at the cost of larger and more compute-in…
Next-Generation Local Time Stepping for the ADER-DG Finite Element Method
Alexander Breuer, Alexander Heinecke
High-frequency ground motion simulations pose a grand challenge in computational seismology. Two main factors drive this challenge. First, to account for higher frequencies, we hav…
Efficient and Generic 1D Dilated Convolution Layer for Deep Learning
Narendra Chaudhary, Sanchit Misra, Dhiraj Kalamkar +5
Convolutional neural networks (CNNs) have found many applications in tasks involving two-dimensional (2D) data, such as image classification and image processing. Therefore, 2D con…
DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networks
Vasimuddin Md, Sanchit Misra, Guixiang Ma +6
Full-batch training on Graph Neural Networks (GNN) to learn the structure of large graphs is a critical problem that needs to scale to hundreds of compute nodes to be feasible. It…
Tensor Processing Primitives: A Programming Abstraction for Efficiency and Portability in Deep Learning & HPC Workloads
Evangelos Georganas, Dhiraj Kalamkar, Sasikanth Avancha +16
During the past decade, novel Deep Learning (DL) algorithms, workloads and hardware have been developed to tackle a wide range of problems. Despite the advances in workload and har…
PolyDL: Polyhedral Optimizations for Creation of High Performance DL primitives
Sanket Tavarageri, Alexander Heinecke, Sasikanth Avancha +3
Deep Neural Networks (DNNs) have revolutionized many aspects of our lives. The use of DNNs is becoming ubiquitous including in softwares for image recognition, speech recognition,…
Optimizing Deep Learning Recommender Systems' Training On CPU Cluster Architectures
Dhiraj Kalamkar, Evangelos Georganas, Sudarshan Srinivasan +3
During the last two years, the goal of many researchers has been to squeeze the last bit of performance out of HPC system for AI tasks. Often this discussion is held in the context…
PolyScientist: Automatic Loop Transformations Combined with Microkernels for Optimization of Deep Learning Primitives
Sanket Tavarageri, Alexander Heinecke, Sasikanth Avancha +3
At the heart of deep learning training and inferencing are computationally intensive primitives such as convolutions which form the building blocks of deep neural networks. Researc…
Training Neural Machine Translation (NMT) Models using Tensor Train Decomposition on TensorFlow (T3F)
Amelia Drew, Alexander Heinecke
We implement a Tensor Train layer in the TensorFlow Neural Machine Translation (NMT) model using the t3f library. We perform training runs on the IWSLT English-Vietnamese '15 and W…
High-Performance Deep Learning via a Single Building Block
Evangelos Georganas, Kunal Banerjee, Dhiraj Kalamkar +6
Deep learning (DL) is one of the most prominent branches of machine learning. Due to the immense computational cost of DL workloads, industry and academia have developed DL librari…
A Study of BFLOAT16 for Deep Learning Training
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi +16
This paper presents the first comprehensive empirical study demonstrating the efficacy of the Brain Floating Point (BFLOAT16) half-precision format for Deep Learning training acros…
Leveraging the bfloat16 Artificial Intelligence Datatype For Higher-Precision Computations
Greg Henry, Ping Tak Peter Tang, Alexander Heinecke
In recent years fused-multiply-add (FMA) units with lower-precision multiplications and higher-precision accumulation have proven useful in machine learning/artificial intelligence…
ISA Mapper: A Compute and Hardware Agnostic Deep Learning Compiler
Matthew Sotoudeh, Anand Venkat, Michael Anderson +3
Domain specific accelerators present new challenges and opportunities for code generation onto novel instruction sets, communication fabrics, and memory architectures. In this pape…
Anatomy Of High-Performance Deep Learning Convolutions On SIMD Architectures
Evangelos Georganas, Sasikanth Avancha, Kunal Banerjee +4
Convolution layers are prevalent in many classes of deep neural networks, including Convolutional Neural Networks (CNNs) which provide state-of-the-art results for tasks like image…
Mixed Precision Training of Convolutional Neural Networks using Integer Operations
Dipankar Das, Naveen Mellempudi, Dheevatsa Mudigere +14
The state-of-the-art (SOTA) for mixed precision training is dominated by variants of low precision floating point operations, and in particular, FP16 accumulating into FP32 Micikev…
ls1 mardyn: The massively parallel molecular dynamics code for large systems
Christoph Niethammer, Stefan Becker, Martin Bernreuther +9
The molecular dynamics simulation code ls1 mardyn is presented. It is a highly scalable code, optimized for massively parallel execution on supercomputing architectures, and curren…