8 papers
Seen, Said, or Forgotten? A Causal Audit of Visual KV Memory Across Dialog Turns
Hong Chen, Kang Chen, Yuxuan Fan +4
Stateful multimodal assistants encode an image once but may answer questions about it many turns later. Attention-guided visual-KV eviction assumes that evidence irrelevant now wil…
GoQuant: Geometric Orthogonal Residual Projection for Multiplier-Free Power-of-Two Transformer Quantization
Maoyang Xiang, Tao Luo, Bo Wang
The deployment of Large Language Models (LLMs) and Vision Transformers (ViTs) on edge devices is significantly constrained by memory capacity and the critical timing bottlenecks in…
NestedKV: Nested Memory Routing for Long-Context KV Cache Compression
Hong Chen, Xiang Liu, Yubo Gao +5
Long-context language models are limited by the memory footprint of the key-value (KV) cache. Existing training-free KV compression methods usually rank tokens by one importance si…
Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training
Wenjie Zhou, Bohan Wang, Hongtao Zhang +3
Model merging has emerged as a lightweight paradigm for enhancing Large Language Models (LLMs), yet its underlying mechanisms remain poorly understood. In this work, we analyze lat…
SSR: Speculative Parallel Scaling Reasoning in Test-time
Yuanlin Chu, Bo Wang, Xiang Liu +3
Large language models (LLMs) have achieved impressive results on multi-step mathematical reasoning, yet at the cost of high computational overhead. This challenge is particularly a…
Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models
Bo Wang, Junzhuo Li, Hong Chen +3
Mixture-of-Experts (MoE) architectures decouple model capacity from per-token computation, enabling scaling beyond the computational limits imposed by dense scaling laws. Yet how M…