6 papers
SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification
Kanghoon Yoon, Minsub Kim, Sungjae Lee +6
Speculative decoding accelerates LLM inference by verifying candidate tokens from a draft model against a larger target model. Recent judge decoding boosts this process by relaxing…
Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention
Jeongin Bae, Baeseong Park, Gunho Park +7
Transformer attention is typically implemented using softmax normalization, which enforces attention weights with unit sum normalization. While effective in many settings, this con…
An Inquiry into Datacenter TCO for LLM Inference with FP8
Jiwoo Kim, Joonhyung Lee, Gunho Park +4
As large language models (LLMs) continue to scale, the high power consumption of AI accelerators in datacenters presents significant challenges, substantially increasing the total…
To FP8 and Back Again: Quantifying Reduced Precision Effects on LLM Training Stability
Joonhyung Lee, Jeongin Bae, Byeongwook Kim +2
The massive computational costs associated with large language model (LLM) pretraining have spurred great interest in reduced-precision floating-point representations to accelerate…
Debunking the CUDA Myth Towards GPU-based AI Systems
Yunjae Lee, Juntaek Lim, Jehyeon Bang +10
This paper presents a comprehensive evaluation of Intel Gaudi NPUs as an alternative to NVIDIA GPUs, which is currently the de facto standard in AI system design. First, we create…
Faster Inference of LLMs using FP8 on the Intel Gaudi
Joonhyung Lee, Shmulik Markovich-Golan, Daniel Ohayon +9
Low-precision data types are essential in modern neural networks during both training and inference as they enhance throughput and computational capacity by better exploiting avail…