7 papers
Attention Editing: A Versatile Framework for Cross-Architecture Attention Conversion
Zhen Cheng, Hao-Bo Yang, Wan-Yi Huang +1
Key-Value (KV) cache memory and bandwidth increasingly dominate large language model inference cost in long-context and long-generation regimes. Architectures such as multi-head la…
CLASH: Collaborative Large-Small Hierarchical Framework for Continuous Vision-and-Language Navigation
Liuyi Wang, Zongtao He, Jinlong Li +6
Vision-and-Language Navigation (VLN) requires robots to follow natural language instructions and navigate complex environments without prior maps. While recent vision-language larg…
RADAR: Accelerating Large Language Model Inference With RL-Based Dynamic Draft Trees
Junjie Ma, Jinlong Li
Inference with modern Large Language Models (LLMs) is expensive and slow, and speculative sampling has emerged as an effective solution to this problem, however, the number of the…
ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Models
Manlai Liang, Mandi Liu, Jiangzhou Ji +4
Large Language Models (LLMs) have demonstrated success across many benchmarks. However, they still exhibit limitations in long-context scenarios, primarily due to their short effec…
LagKV: Lag-Relative Information of the KV Cache Tells Which Tokens Are Important
Manlai Liang, JiaMing Zhang, Xiong Li +1
The increasing size of the Key-Value (KV) cache during the Large Language Models long-context inference is the main obstacle for its balance between the deployment cost and task ac…
Lag-Relative Sparse Attention In Long Context Training
Manlai Liang, Wanyi Huang, Mandi Liu +2
Large Language Models (LLMs) have made significant strides in natural language processing and generation, yet their ability to handle long-context input remains constrained by the…