7 papers
Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention
Jeongin Bae, Baeseong Park, Gunho Park +7
Transformer attention is typically implemented using softmax normalization, which enforces attention weights with unit sum normalization. While effective in many settings, this con…
Training-free Dropout Sampling for Semantic Token Acceptance in Speculative Decoding
Jeongtae Lee, Minjung Jo, Hyunjoon Jeong +5
Speculative decoding accelerates large language model inference by proposing tokens with a lightweight draft model and selectively accepting them using a target model. This work in…
AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs
Gunho Park, Jeongin Bae, Beomseok Kwon +3
The deployment of large language models (LLMs) is increasingly constrained by memory and latency bottlenecks, motivating the need for quantization techniques that flexibly balance…
CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs
Gunho Park, Jeongin Bae, Byeongwook Kim +5
Weight-only quantization is widely used to mitigate the memory-bound nature of LLM inference. Codebook-based methods extend this trend by achieving strong accuracy in the extremely…
An Inquiry into Datacenter TCO for LLM Inference with FP8
Jiwoo Kim, Joonhyung Lee, Gunho Park +4
As large language models (LLMs) continue to scale, the high power consumption of AI accelerators in datacenters presents significant challenges, substantially increasing the total…
Faster Inference of LLMs using FP8 on the Intel Gaudi
Joonhyung Lee, Shmulik Markovich-Golan, Daniel Ohayon +9
Low-precision data types are essential in modern neural networks during both training and inference as they enhance throughput and computational capacity by better exploiting avail…