1 citations · 2 across the 7 of their papers we have counts for
8 papers
LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention
Zhaohui Wang, Zhixin Pan, Fanxu Meng +1
Sparse attention reduces core-attention computation, but its indexers still incur repeated selection work and per-layer key-cache storage. Reusing selected indices across layers re…
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
Yufei Xu, Fanxu Meng, Fan Jiang +11
Token-level sparse attention mechanisms, exemplified by DeepSeek Sparse Attention (DSA), achieve fine-grained key selection by scoring every historical key for each query through a…
LooGLE v2: Are LLMs Ready for Real World Long Dependency Challenges?
Ziyuan He, Yuxuan Wang, Jiaqi Li +2
Large language models (LLMs) are equipped with increasingly extended context windows recently, yet their long context understanding capabilities over long dependency tasks remain f…
TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference
Xiaojuan Tang, Fanxu Meng, Pingzhi Tang +4
Multi-Head Latent Attention (MLA), introduced in DeepSeek-V2, compresses key-value states into a low-rank latent vector, caching only this vector to reduce memory. In tensor parall…
LIFT: A Novel Framework for Enhancing Long-Context Understanding of LLMs via Long Input Fine-Tuning
Yansheng Mao, Yufei Xu, Jiaqi Li +5
Long-context understanding remains challenging for LLMs due to limited context windows. This paper introduces Long Input Fine-Tuning (LIFT), a framework that improves the long-cont…
LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning
Yansheng Mao, Jiaqi Li, Fanxu Meng +3
Long context understanding remains challenging for large language models due to their limited context windows. This paper introduces Long Input Fine-Tuning (LIFT) for long context…
Mars: Situated Inductive Reasoning in an Open-World Environment
Xiaojuan Tang, Jiaqi Li, Yitao Liang +3
Large Language Models (LLMs) trained on massive corpora have shown remarkable success in knowledge-intensive tasks. Yet, most of them rely on pre-stored knowledge. Inducing new gen…
LooGLE: Can Long-Context Language Models Understand Long Contexts?
Jiaqi Li, Mengmeng Wang, Zilong Zheng +1
Large language models (LLMs), despite their impressive performance in various language tasks, are typically limited to processing texts within context-window size. This limitation…