124 citations · 399 across the 12 of their papers we have counts for
13 papers
Exploring FPGA designs for MX and beyond
Ebby Samson, Naveen Mellempudi, Wayne Luk +1
A number of companies recently worked together to release the new Open Compute Project MX standard for low-precision computation, aimed at efficient neural network implementation.…
Efficient Post-training Quantization with FP8 Formats
Haihao Shen, Naveen Mellempudi, Xin He +3
Recent advances in deep learning methods such as LLMs and Diffusion models have created a need for improved quantization methods that can meet the computational demands of these mo…
AUTOSPARSE: Towards Automated Sparse Training of Deep Neural Networks
Abhisek Kundu, Naveen K. Mellempudi, Dharma Teja Vooturi +2
Sparse training is emerging as a promising avenue for reducing the computational cost of training neural networks. Several recent studies have proposed pruning methods using learna…
FP8 Formats for Deep Learning
Paulius Micikevicius, Dusan Stosic, Neil Burgess +12
FP8 is a natural progression for accelerating deep learning training inference beyond the 16-bit formats common in modern processors. In this paper we propose an 8-bit floating poi…
K-TanH: Efficient TanH For Deep Learning
Abhisek Kundu, Alex Heinecke, Dhiraj Kalamkar +7
We propose K-TanH, a novel, highly accurate, hardware efficient approximation of popular activation function TanH for Deep Learning. K-TanH consists of parameterized low-precision…
High Performance Scalable FPGA Accelerator for Deep Neural Networks
Sudarshan Srinivasan, Pradeep Janedula, Saurabh Dhoble +7
Low-precision is the first order knob for achieving higher Artificial Intelligence Operations (AI-TOPS). However the algorithmic space for sub-8-bit precision compute is diverse, w…
A Study of BFLOAT16 for Deep Learning Training
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi +16
This paper presents the first comprehensive empirical study demonstrating the efficacy of the Brain Floating Point (BFLOAT16) half-precision format for Deep Learning training acros…
Mixed Precision Training With 8-bit Floating Point
Naveen Mellempudi, Sudarshan Srinivasan, Dipankar Das +1
Reduced precision computation for deep neural networks is one of the key areas addressing the widening compute gap driven by an exponential growth in model size. In recent years, d…
Mixed Precision Training of Convolutional Neural Networks using Integer Operations
Dipankar Das, Naveen Mellempudi, Dheevatsa Mudigere +14
The state-of-the-art (SOTA) for mixed precision training is dominated by variants of low precision floating point operations, and in particular, FP16 accumulating into FP32 Micikev…
On Scale-out Deep Learning Training for Cloud and HPC
Srinivas Sridharan, Karthikeyan Vaidyanathan, Dhiraj Kalamkar +8
The exponential growth in use of large deep neural networks has accelerated the need for training these deep neural networks in hours or even minutes. This can only be achieved thr…
Ternary Residual Networks
Abhisek Kundu, Kunal Banerjee, Naveen Mellempudi +4
Sub-8-bit representation of DNNs incur some discernible loss of accuracy despite rigorous (re)training at low-precision. Such loss of accuracy essentially makes them equivalent to…
Ternary Neural Networks with Fine-Grained Quantization
Naveen Mellempudi, Abhisek Kundu, Dheevatsa Mudigere +3
We propose a novel fine-grained quantization (FGQ) method to ternarize pre-trained full precision models, while also constraining activations to 8 and 4-bits. Using this method, we…
Mixed Low-precision Deep Learning Inference using Dynamic Fixed Point
Naveen Mellempudi, Abhisek Kundu, Dipankar Das +2
We propose a cluster-based quantization method to convert pre-trained full precision weights into ternary weights with minimal impact on the accuracy. In addition, we also constrai…