11 papers
Fitting Is Not Enough: Smoothness in Extremely Quantized LLMs
Yuzhuang Xu, Xu Han, Yuxuan Li +2
Large language models (LLMs) achieve strong performance but incur high deployment costs, motivating extremely low-bit but lossy quantization. Existing quantization algorithms mainl…
ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs
Yuzhuang Xu, Xu Han, Yuxuan Li +1
Although existing frameworks for large language model (LLM) inference on CPUs are mature, they fail to fully exploit the computation potential of many-core CPU platforms. Many-core…
HUOZIIME: An On-Device LLM-enhanced Input Method for Deep Personalization
Baocai Shan, Yuzhuang Xu, Wanxiang Che
Mobile input method editors (IMEs) are the primary interface for text input, yet they remain constrained to manual typing and struggle to produce personalized text. While lightweig…
Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction
Yijun Liu, Yixuan Wang, Yuzhuang Xu +4
Large language models (LLMs) utilize key-value (KV) cache to store historical information during sequence processing. The size of KV cache grows linearly as the length of the seque…
CAMERA: Multi-Matrix Joint Compression for MoE Models via Micro-Expert Redundancy Analysis
Yuzhuang Xu, Xu Han, Yuanchi Zhang +5
Large Language Models (LLMs) with Mixture-of-Experts (MoE) architectures are distinguished by their strong performance scaling with increasing parameters across a wide range of tas…
Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query
Yixuan Wang, Shiyu Ji, Yijun Liu +4
Large language models (LLMs) rely on key-value cache (KV cache) to accelerate decoding by reducing redundant computations. However, the KV cache memory usage grows substantially wi…