77 citations · 349 across the 17 of their papers we have counts for
17 papers
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh +4
LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to m…
Learned Best-Effort LLM Serving
Siddharth Jha, Coleman Hooper, Xiaoxuan Liu +2
Many applications must provide low-latency LLM service to users or risk unacceptable user experience. However, over-provisioning resources to serve fluctuating request patterns is…
SPEED: Speculative Pipelined Execution for Efficient Decoding
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh +4
Generative Large Language Models (LLMs) based on the Transformer architecture have recently emerged as a dominant foundation model for a wide range of Natural Language Processing t…
SqueezeLLM: Dense-and-Sparse Quantization
Sehoon Kim, Coleman Hooper, Amir Gholami +5
Generative Large Language Models (LLMs) have demonstrated remarkable results for a wide range of tasks. However, deploying these models for inference has been a significant challen…
Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling
Xiuying Wei, Yunchen Zhang, Yuhang Li +4
Post-training quantization~(PTQ) of transformer language models faces significant challenges due to the existence of detrimental outliers in activations. We observe that these outl…
Full Stack Optimization of Transformer Inference: a Survey
Sehoon Kim, Coleman Hooper, Thanakul Wattanawong +9
Recent advances in state-of-the-art DNN architecture design have been moving toward Transformer models. These models achieve superior accuracy across a wide range of applications.…
Speculative Decoding with Big Little Decoder
Sehoon Kim, Karttikeya Mangalam, Suhong Moon +4
The recent emergence of Large Language Models based on the Transformer architecture has enabled dramatic advancements in the field of Natural Language Processing. However, these mo…
BigColor: Colorization using a Generative Color Prior for Natural Images
Geonung Kim, Kyoungkook Kang, Seongtae Kim +5
For realistic and vivid colorization, generative priors have recently been exploited. However, such generative priors often fail for in-the-wild complex images due to their limited…
Squeezeformer: An Efficient Transformer for Automatic Speech Recognition
Sehoon Kim, Amir Gholami, Albert Shaw +5
The recently proposed Conformer model has become the de facto backbone model for various downstream speech tasks based on its hybrid attention-convolution architecture that capture…
A Fast Post-Training Pruning Framework for Transformers
Woosuk Kwon, Sehoon Kim, Michael W. Mahoney +3
Pruning is an effective way to reduce the huge inference cost of Transformer models. However, prior work on pruning Transformers requires retraining the models. This can add high t…
Terra: Imperative-Symbolic Co-Execution of Imperative Deep Learning Programs
Taebum Kim, Eunji Jeong, Geon-Woo Kim +4
Imperative programming allows users to implement their deep neural networks (DNNs) easily and has become an essential part of recent deep learning (DL) frameworks. Recently, severa…
A Static Analyzer for Detecting Tensor Shape Errors in Deep Neural Network Training Code
Ho Young Jhoo, Sehoon Kim, Woosung Song +3
We present an automatic static analyzer PyTea that detects tensor-shape errors in PyTorch code. The tensor-shape error is critical in the deep neural net code; much of the training…
Applications and Techniques for Fast Machine Learning in Science
Allison McCarn Deiana, Nhan Tran, Joshua Agar +84
In this community review report, we discuss applications and techniques for fast machine learning (ML) in science -- the concept of integrating power ML methods into the real-time…
Learned Token Pruning for Transformers
Sehoon Kim, Sheng Shen, David Thorsley +4
Deploying transformer models in practice is challenging due to their inference cost, which scales quadratically with input sequence length. To address this, we present a novel Lear…
Integer-only Zero-shot Quantization for Efficient Speech Recognition
Sehoon Kim, Amir Gholami, Zhewei Yao +7
End-to-end neural network models achieve improved performance on various automatic speech recognition (ASR) tasks. However, these models perform poorly on edge hardware due to larg…
A Survey of Quantization Methods for Efficient Neural Network Inference
Amir Gholami, Sehoon Kim, Zhen Dong +3
As soon as abstract mathematical computations were adapted to computation on digital computers, the problem of efficient representation, manipulation, and communication of the nume…
I-BERT: Integer-only BERT Quantization
Sehoon Kim, Amir Gholami, Zhewei Yao +2
Transformer based models, like BERT and RoBERTa, have achieved state-of-the-art results in many Natural Language Processing tasks. However, their memory footprint, inference latenc…