5 papers
TRACE: Unlocking Effective CXL Bandwidth via Lossless Compression and Precision Scaling
Rui Xie, Asad Ul Haq, Yunhua Fang +4
LLM inference is increasingly limited by memory bandwidth, and the bottleneck worsens at long context as the KV cache grows. CXL memory adds capacity to offload weights and KV, but…
Making Strong Error-Correcting Codes Work Effectively for HBM in AI Inference
Rui Xie, Yunhua Fang, Asad Ul Haq +5
LLM inference is increasingly memory bound, and HBM cost per GB dominates system cost. Current HBM stacks include short on-die ECC that tightens binning, raises price, and fixes re…
Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System
Yunhua Fang, Rui Xie, Asad Ul Haq +6
Large Language Model (LLM) inference is increasingly constrained by memory bandwidth, with frequent access to the key-value (KV) cache dominating data movement. While attention spa…
Breaking the HBM Bit Cost Barrier: Domain-Specific ECC for AI Inference Infrastructure
Rui Xie, Asad Ul Haq, Yunhua Fang +5
High-Bandwidth Memory (HBM) delivers exceptional bandwidth and energy efficiency for AI workloads, but its high cost per bit, driven in part by stringent on-die reliability require…
Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
Rui Xie, Asad Ul Haq, Linsen Ma +4
The efficiency of Large Language Model~(LLM) inference is often constrained by substantial memory bandwidth and capacity demands. Existing techniques, such as pruning, quantization…