14 papers
SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification
Kanghoon Yoon, Minsub Kim, Sungjae Lee +6
Speculative decoding accelerates LLM inference by verifying candidate tokens from a draft model against a larger target model. Recent judge decoding boosts this process by relaxing…
SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving
Sunghyeon Woo, Ahreum Seo, Jaegwang Lee +6
In multi-model LLM serving, decode execution remains inefficient due to model-specific resource partitioning: since cross-model batching is not possible, memory-bound decoding ofte…
ICaRus: Identical Cache Reuse for Efficient Multi Model Inference
Sunghyeon Woo, Jaeeun Kil, Hoseung Kim +9
Multi model inference has recently emerged as a prominent paradigm, particularly in the development of agentic AI systems. However, in such scenarios, each model must maintain its…
Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention
Jeongin Bae, Baeseong Park, Gunho Park +7
Transformer attention is typically implemented using softmax normalization, which enforces attention weights with unit sum normalization. While effective in many settings, this con…
PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving
Sunghyeon Woo, Hoseung Kim, Sunghwan Shim +8
Multi-agent systems increasingly orchestrate multiple specialized language models to solve complex real-world problems, often invoking them over a shared context. This execution pa…
AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs
Gunho Park, Jeongin Bae, Beomseok Kwon +3
The deployment of large language models (LLMs) is increasingly constrained by memory and latency bottlenecks, motivating the need for quantization techniques that flexibly balance…