77 citations · 349 across the 17 of their papers we have counts for
5 papers · 1 filter
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh +4
LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to m…
Learned Best-Effort LLM Serving
Siddharth Jha, Coleman Hooper, Xiaoxuan Liu +2
Many applications must provide low-latency LLM service to users or risk unacceptable user experience. However, over-provisioning resources to serve fluctuating request patterns is…
Terra: Imperative-Symbolic Co-Execution of Imperative Deep Learning Programs
Taebum Kim, Eunji Jeong, Geon-Woo Kim +4
Imperative programming allows users to implement their deep neural networks (DNNs) easily and has become an essential part of recent deep learning (DL) frameworks. Recently, severa…
A Static Analyzer for Detecting Tensor Shape Errors in Deep Neural Network Training Code
Ho Young Jhoo, Sehoon Kim, Woosung Song +3
We present an automatic static analyzer PyTea that detects tensor-shape errors in PyTorch code. The tensor-shape error is critical in the deep neural net code; much of the training…
Applications and Techniques for Fast Machine Learning in Science
Allison McCarn Deiana, Nhan Tran, Joshua Agar +84
In this community review report, we discuss applications and techniques for fast machine learning (ML) in science -- the concept of integrating power ML methods into the real-time…