Showing cs.LGShow all
2 papers · 1 filter
cs.LG2026
SAKI-RoPE: Score-Aware Low-Rank Key Indexing with RoPE-Aware Covariance and Random-Matrix Noise Analysis
Lin Zhang
Existing low-rank KV-cache indexes preserve model weights or key variance, not the attention scores that determine retrieval. We derive the expected score distortion of key-side ra…
cs.LG2025
BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference
Zewen Jin, Shengnan Wang, Jiaan Zhu +5
The Mixture-of-Experts (MoE) structure scales the Transformer-based large language models (LLMs) and improves their performance with only the sub-linear increase in computation res…