4 papers
OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
Can Xiao, Sukmin Cho, Junbong We +7
Large language model (LLM) inference serving is increasingly constrained by memory rather than compute. As long-context and long-form reasoning workloads become more prevalent, the…
Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
Zifan He, Rui Ma, Yizhou Sun +1
Modern large language models (LLMs) increasingly depends on efficient long-context processing and generation mechanisms, including sparse attention, retrieval-augmented generation…
LUMINA: LLM-Guided GPU Architecture Exploration via Bottleneck Analysis
Tao Zhang, Rui Ma, Shuotao Xu +2
GPU design space exploration (DSE) for modern AI workloads, such as Large-Language Model (LLM) inference, is challenging because of GPUs' vast, multi-modal design spaces, high simu…
TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge
Zhirui Huang, Rui Ma, Shijie Cao +5
Ternary quantization has emerged as a powerful technique for reducing both computational and memory footprint of large language models (LLM), enabling efficient real-time inference…