6 papers
Spexis: Speculative Lookahead Scheduling for LLM Inference
Hyungyu Jung, Jaehyeok Yu, Hoonseo Choi +3
Spexis is a multi-GPU LLM inference framework that improves the efficiency of pipeline and tensor parallelism through speculative parallelism. Rather than using speculative decodin…
Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimate
Jaemin Kim, Sungkyun Kim, Junyeol Lee +1
Large Language Models (LLMs) are widely used across many domains, but their scale makes deployment challenging. Post-Training Quantization (PTQ) reduces memory footprint without re…
FlexiQ: Adaptive Mixed-Precision Quantization for Latency/Accuracy Trade-Offs in Deep Neural Networks
Jaemin Kim, Hongjun Um, Sungkyun Kim +2
Neural networks commonly execute on hardware accelerators such as NPUs and GPUs for their size and computation overhead. These accelerators are costly and it is hard to scale their…
Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding
Sungkyun Kim, Jaemin Kim, Dogyung Yoon +3
LLMs have low GPU efficiency and high latency due to autoregressive decoding. Speculative decoding (SD) mitigates this using a small draft model to speculatively generate multiple…
Benchmarks Underestimate the Readiness of Multi-lingual Dialogue Agents
Andrew H. Lee, Sina J. Semnani, Galo Castillo-López +16
Creating multilingual task-oriented dialogue (TOD) agents is challenging due to the high cost of training data acquisition. Following the research trend of improving training data…
ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference
Hyungjun Oh, Kihong Kim, Jaemin Kim +4
This paper presents ExeGPT, a distributed system designed for constraint-aware LLM inference. ExeGPT finds and runs with an optimal execution schedule to maximize inference through…