7 papers
Seen, Said, or Forgotten? A Causal Audit of Visual KV Memory Across Dialog Turns
Hong Chen, Kang Chen, Yuxuan Fan +4
Stateful multimodal assistants encode an image once but may answer questions about it many turns later. Attention-guided visual-KV eviction assumes that evidence irrelevant now wil…
NestedKV: Nested Memory Routing for Long-Context KV Cache Compression
Hong Chen, Xiang Liu, Yubo Gao +5
Long-context language models are limited by the memory footprint of the key-value (KV) cache. Existing training-free KV compression methods usually rank tokens by one importance si…
GoQuant: Geometric Orthogonal Residual Projection for Multiplier-Free Power-of-Two Transformer Quantization
Maoyang Xiang, Tao Luo, Bo Wang
The deployment of Large Language Models (LLMs) and Vision Transformers (ViTs) on edge devices is significantly constrained by memory capacity and the critical timing bottlenecks in…
Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models
Bo Wang, Junzhuo Li, Hong Chen +3
Mixture-of-Experts (MoE) architectures decouple model capacity from per-token computation, enabling scaling beyond the computational limits imposed by dense scaling laws. Yet how M…
Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation
Junzhuo Li, Bo Wang, Xiuze Zhou +1
Mixture-of-Experts (MoE) models offer immense capacity via sparsely gated expert subnetworks, yet adapting them to multiple domains without catastrophic forgetting remains an open…
Decoding Knowledge Attribution in Mixture-of-Experts: A Framework of Basic-Refinement Collaboration and Efficiency Analysis
Junzhuo Li, Bo Wang, Xiuze Zhou +3
The interpretability of Mixture-of-Experts (MoE) models, especially those with heterogeneous designs, remains underexplored. Existing attribution methods for dense models fail to c…
SSR: Speculative Parallel Scaling Reasoning in Test-time
Yuanlin Chu, Bo Wang, Xiang Liu +3
Large language models (LLMs) have achieved impressive results on multi-step mathematical reasoning, yet at the cost of high computational overhead. This challenge is particularly a…