7 papers
Five-Minute Rule 40 Years Later: A First-Principles Revisit for Modern Memory Hierarchy
Tong Zhang, Vikram Sharma Mailthody, Fei Sun +7
In 1987, Jim Gray and Gianfranco Putzolu introduced the five-minute rule, a simple, storage-memory-economics-based heuristic for deciding when data should live in DRAM rather than…
TRACE: Unlocking Effective CXL Bandwidth via Lossless Compression and Precision Scaling
Rui Xie, Asad Ul Haq, Yunhua Fang +4
LLM inference is increasingly limited by memory bandwidth, and the bottleneck worsens at long context as the KV cache grows. CXL memory adds capacity to offload weights and KV, but…
Making Strong Error-Correcting Codes Work Effectively for HBM in AI Inference
Rui Xie, Yunhua Fang, Asad Ul Haq +5
LLM inference is increasingly memory bound, and HBM cost per GB dominates system cost. Current HBM stacks include short on-die ECC that tightens binning, raises price, and fixes re…
Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System
Yunhua Fang, Rui Xie, Asad Ul Haq +6
Large Language Model (LLM) inference is increasingly constrained by memory bandwidth, with frequent access to the key-value (KV) cache dominating data movement. While attention spa…
Breaking the HBM Bit Cost Barrier: Domain-Specific ECC for AI Inference Infrastructure
Rui Xie, Asad Ul Haq, Yunhua Fang +5
High-Bandwidth Memory (HBM) delivers exceptional bandwidth and energy efficiency for AI workloads, but its high cost per bit, driven in part by stringent on-die reliability require…
Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
Rui Xie, Asad Ul Haq, Linsen Ma +4
The efficiency of Large Language Model~(LLM) inference is often constrained by substantial memory bandwidth and capacity demands. Existing techniques, such as pruning, quantization…