148 citations · 428 across the 18 of their papers we have counts for
18 papers
Scalable Pretraining of Large Mixture of Experts Language Models on Aurora Super Computer
Dharma Teja Vooturi, Dhiraj Kalamkar, Dipankar Das +1
Pretraining Large Language Models (LLMs) from scratch requires massive amount of compute. Aurora super computer is an ExaScale machine with 127,488 Intel PVC (Ponte Vechio) GPU til…
Pushing the Envelope of LLM Inference with Ultra-Low-Bit Quantized Models
Evangelos Georganas, Dhiraj Kalamkar, Alexander Heinecke +1
The advent of ultra-low-bit LLM models, approaching the perplexity and task accuracy of their full precision counterparts, is ushering in a new era of LLM inference. While these ad…
ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts
Evangelos Georganas, Dhiraj Kalamkar, Alexander Kozlov +1
Speculative decoding (SD) has emerged as a method to accelerate LLM inference without sacrificing any accuracy over the 16-bit model inference. In a typical SD setup, the idea is t…
DQRM: Deep Quantized Recommendation Models
Yang Zhou, Zhen Dong, Ellick Chan +3
Large-scale recommendation models are currently the dominant workload for many large Internet companies. These recommenders are characterized by massive embedding tables that are s…
Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures
Evangelos Georganas, Dhiraj Kalamkar, Kirill Voronin +5
During the past decade, Deep Learning (DL) algorithms, programming systems and hardware have converged with the High Performance Computing (HPC) counterparts. Nevertheless, the pro…
Efficient and Generic 1D Dilated Convolution Layer for Deep Learning
Narendra Chaudhary, Sanchit Misra, Dhiraj Kalamkar +5
Convolutional neural networks (CNNs) have found many applications in tasks involving two-dimensional (2D) data, such as image classification and image processing. Therefore, 2D con…
DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networks
Vasimuddin Md, Sanchit Misra, Guixiang Ma +6
Full-batch training on Graph Neural Networks (GNN) to learn the structure of large graphs is a critical problem that needs to scale to hundreds of compute nodes to be feasible. It…
Tensor Processing Primitives: A Programming Abstraction for Efficiency and Portability in Deep Learning & HPC Workloads
Evangelos Georganas, Dhiraj Kalamkar, Sasikanth Avancha +16
During the past decade, novel Deep Learning (DL) algorithms, workloads and hardware have been developed to tackle a wide range of problems. Despite the advances in workload and har…
Optimizing Deep Learning Recommender Systems' Training On CPU Cluster Architectures
Dhiraj Kalamkar, Evangelos Georganas, Sudarshan Srinivasan +3
During the last two years, the goal of many researchers has been to squeeze the last bit of performance out of HPC system for AI tasks. Often this discussion is held in the context…
K-TanH: Efficient TanH For Deep Learning
Abhisek Kundu, Alex Heinecke, Dhiraj Kalamkar +7
We propose K-TanH, a novel, highly accurate, hardware efficient approximation of popular activation function TanH for Deep Learning. K-TanH consists of parameterized low-precision…
High-Performance Deep Learning via a Single Building Block
Evangelos Georganas, Kunal Banerjee, Dhiraj Kalamkar +6
Deep learning (DL) is one of the most prominent branches of machine learning. Due to the immense computational cost of DL workloads, industry and academia have developed DL librari…
A Study of BFLOAT16 for Deep Learning Training
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi +16
This paper presents the first comprehensive empirical study demonstrating the efficacy of the Brain Floating Point (BFLOAT16) half-precision format for Deep Learning training acros…
Anatomy Of High-Performance Deep Learning Convolutions On SIMD Architectures
Evangelos Georganas, Sasikanth Avancha, Kunal Banerjee +4
Convolution layers are prevalent in many classes of deep neural networks, including Convolutional Neural Networks (CNNs) which provide state-of-the-art results for tasks like image…
Mixed Precision Training of Convolutional Neural Networks using Integer Operations
Dipankar Das, Naveen Mellempudi, Dheevatsa Mudigere +14
The state-of-the-art (SOTA) for mixed precision training is dominated by variants of low precision floating point operations, and in particular, FP16 accumulating into FP32 Micikev…
On Scale-out Deep Learning Training for Cloud and HPC
Srinivas Sridharan, Karthikeyan Vaidyanathan, Dhiraj Kalamkar +8
The exponential growth in use of large deep neural networks has accelerated the need for training these deep neural networks in hours or even minutes. This can only be achieved thr…
MILC staggered conjugate gradient performance on Intel KNL
Carleton DeTar, Douglas Doerfler, Steven Gottlieb +4
We review our work done to optimize the staggered conjugate gradient (CG) algorithm in the MILC code for use with the Intel Knights Landing (KNL) architecture. KNL is the second ge…
Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
Dipankar Das, Sasikanth Avancha, Dheevatsa Mudigere +5
We design and implement a distributed multinode synchronous SGD algorithm, without altering hyper parameters, or compressing data, or altering algorithmic behavior. We perform a de…
Lattice QCD with Domain Decomposition on Intel Xeon Phi Co-Processors
Simon Heybrock, Bálint Joó, Dhiraj D. Kalamkar +4
The gap between the cost of moving data and the cost of computing continues to grow, making it ever harder to design iterative solvers on extreme-scale architectures. This problem…