4 papers
Is Finer Better? The Limits of Microscaling Formats in Large Language Models
Andrea Fasoli, Monodeep Kar, Chi-Chun Liu +4
Microscaling data formats leverage per-block tensor quantization to enable aggressive model compression with limited loss in accuracy. Unlocking their potential for efficient train…
Making Strong Error-Correcting Codes Work Effectively for HBM in AI Inference
Rui Xie, Yunhua Fang, Asad Ul Haq +5
LLM inference is increasingly memory bound, and HBM cost per GB dominates system cost. Current HBM stacks include short on-die ECC that tightens binning, raises price, and fixes re…
Breaking the HBM Bit Cost Barrier: Domain-Specific ECC for AI Inference Infrastructure
Rui Xie, Asad Ul Haq, Yunhua Fang +5
High-Bandwidth Memory (HBM) delivers exceptional bandwidth and energy efficiency for AI workloads, but its high cost per bit, driven in part by stringent on-die reliability require…
SmartQuant: CXL-based AI Model Store in Support of Runtime Configurable Weight Quantization
Rui Xie, Asad Ul Haq, Linsen Ma +5
Recent studies have revealed that, during the inference on generative AI models such as transformer, the importance of different weights exhibits substantial context-dependent vari…