14 papers
SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification
Kanghoon Yoon, Minsub Kim, Sungjae Lee +6
Speculative decoding accelerates LLM inference by verifying candidate tokens from a draft model against a larger target model. Recent judge decoding boosts this process by relaxing…
SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving
Sunghyeon Woo, Ahreum Seo, Jaegwang Lee +6
In multi-model LLM serving, decode execution remains inefficient due to model-specific resource partitioning: since cross-model batching is not possible, memory-bound decoding ofte…
Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention
Jeongin Bae, Baeseong Park, Gunho Park +7
Transformer attention is typically implemented using softmax normalization, which enforces attention weights with unit sum normalization. While effective in many settings, this con…
Training-free Dropout Sampling for Semantic Token Acceptance in Speculative Decoding
Jeongtae Lee, Minjung Jo, Hyunjoon Jeong +5
Speculative decoding accelerates large language model inference by proposing tokens with a lightweight draft model and selectively accepting them using a target model. This work in…
AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs
Gunho Park, Jeongin Bae, Beomseok Kwon +3
The deployment of large language models (LLMs) is increasingly constrained by memory and latency bottlenecks, motivating the need for quantization techniques that flexibly balance…
CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs
Gunho Park, Jeongin Bae, Byeongwook Kim +5
Weight-only quantization is widely used to mitigate the memory-bound nature of LLM inference. Codebook-based methods extend this trend by achieving strong accuracy in the extremely…