4 papers
Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management
Hyunsun Chung, Taewan Noh, Minji Kim +3
NAND-backed storage offers the capacity needed to scale LLM prefix caching, but its block I/O path incurs CPU cache contention and host-DRAM staging in addition to NAND latency. Ou…
Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloading
Kihyun Kim, Jinwoo Kim, Hyunsun Chung +3
LLM inference is essential for applications like text summarization, translation, and data analysis, but the high cost of GPU instances from Cloud Service Providers (CSPs) like AWS…
Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMs
Hyungwoo Lee, Kihyun Kim, Jinwoo Kim +5
Recent large language models (LLMs) face increasing inference latency as input context length and model size continue to grow. In particular, the retrieval-augmented generation (RA…
DeepVM: Integrating Spot and On-Demand VMs for Cost-Efficient Deep Learning Clusters in the Cloud
Yoochan Kim, Kihyun Kim, Yonghyeon Cho +7
Distributed Deep Learning (DDL), as a paradigm, dictates the use of GPU-based clusters as the optimal infrastructure for training large-scale Deep Neural Networks (DNNs). However,…