4 papers
Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boost
Haojun Xia, Xiaoxia Wu, Jisen Li +12
The KV cache is a dominant memory bottleneck for LLM inference. While 4-bit KV quantization preserves accuracy, 2-bit often degrades it, especially on long-context reasoning. We cl…
Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining
Costin-Andrei Oncescu, Qingyang Wu, Wai Tong Chung +5
An increasing number of LLMs employ Mixture-of-Experts (MoE) architectures where the feed-forward layer is replaced by a pool of experts and each token only activates a small subse…
Imitate Optimal Policy: Prevail and Induce Action Collapse in Policy Gradient
Zhongzhu Zhou, Yibo Yang, Ziyan Chen +7
Policy gradient (PG) methods in reinforcement learning frequently utilize deep neural networks (DNNs) to learn a shared backbone of feature representations used to compute likeliho…
Linguistic Collapse: Neural Collapse in (Large) Language Models
Robert Wu, Vardan Papyan
Neural collapse () is a phenomenon observed in classification tasks where top-layer representations collapse into their class means, which become equinorm, equiangula…