6 papers · 1 filter
Exploring High-Bandwidth Flash for Modern LLM Inference: Opportunities and Challenges
Dowon Son, Yonggon Park, Hyunuk Cho +5
This work investigates the potential benefits and technical challenges of using high-bandwidth flash (HBF) for large language model (LLM) inference. HBF has gained increasing atten…
Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention
Hyungkyu Ham, Junhyeong Bae, Seungheon Lee +2
This paper presents a heterogeneous decode-phase serving system that relocates the KV cache out of GPU memory, motivated by the retrieval-based sparse attention that recent frontie…
Low-overhead General-purpose Near-Data Processing in CXL Memory Expanders
Hyungkyu Ham, Jeongmin Hong, Geonwoo Park +8
Emerging Compute Express Link (CXL) enables cost-efficient memory expansion beyond the local DRAM of processors. While its CXLmem protocol provides minimal latency overhead thro…
ONNXim: A Fast, Cycle-level Multi-core NPU Simulator
Hyungkyu Ham, Wonhyuk Yang, Yunseon Shin +5
As DNNs are widely adopted in various application domains while demanding increasingly higher compute and memory requirements, designing efficient and performant NPUs (Neural Proce…
NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing
Guseul Heo, Sangyeop Lee, Jaehong Cho +6
Modern transformer-based Large Language Models (LLMs) are constructed with a series of decoder blocks. Each block comprises three key components: (1) QKV generation, (2) multi-head…
Bandwidth-Effective DRAM Cache for GPUs with Storage-Class Memory
Jeongmin Hong, Sungjun Cho, Geonwoo Park +3
We propose overcoming the memory capacity limitation of GPUs with high-capacity Storage-Class Memory (SCM) and DRAM cache. By significantly increasing the memory capacity with SCM,…