4 papers
Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System
Yunhua Fang, Rui Xie, Asad Ul Haq +6
Large Language Model (LLM) inference is increasingly constrained by memory bandwidth, with frequent access to the key-value (KV) cache dominating data movement. While attention spa…
ZipCache: A DRAM/SSD Cache with Built-in Transparent Compression
Rui Xie, Linsen Ma, Alex Zhong +2
As a core component in modern data centers, key-value cache provides high-throughput and low-latency services for high-speed data processing. The effectiveness of a key-value cache…
LiteVAR: Compressing Visual Autoregressive Modelling with Efficient Attention and Quantization
Rui Xie, Tianchen Zhao, Zhihang Yuan +5
Visual Autoregressive (VAR) has emerged as a promising approach in image generation, offering competitive potential and performance comparable to diffusion-based models. However, c…
SmartQuant: CXL-based AI Model Store in Support of Runtime Configurable Weight Quantization
Rui Xie, Asad Ul Haq, Linsen Ma +5
Recent studies have revealed that, during the inference on generative AI models such as transformer, the importance of different weights exhibits substantial context-dependent vari…