6 citations · 13 across the 15 of their papers we have counts for
8 papers · 1 filter
LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention
Zhaohui Wang, Zhixin Pan, Fanxu Meng +1
Sparse attention reduces core-attention computation, but its indexers still incur repeated selection work and per-layer key-cache storage. Reusing selected indices across layers re…
DoSA: Dots Sparse Attention
Fanxu Meng, Taosong Fang
Sparse attention visits few keys, yet finding them can still require scoring every key with every indexer head. We introduce DoSA (Dots Sparse Attention), a lossless method for mak…
Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding
Weiye Shi, Fanxu Meng, Muhan Zhang
Multi-head latent attention (MLA) is increasingly important for long-context LLM inference because compact latent states replace the growing key-value (KV) cache and reduce decodin…
MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
Ruijie Zhou, Fanxu Meng, Yufei Xu +4
DeepSeek Sparse Attention (DSA) sets the state of the art for fine-grained inference-time sparse attention by introducing a learned token-wise indexer that scores every prefix toke…
GQLA: Group-Query Latent Attention for Hardware-Adaptive Decoding
Fanxu Meng, Zhixin Pan
Decoding efficiency depends on KV-cache traffic, computation, and parallelism. Multi-head Latent Attention (MLA) offers absorbed and head-expanded forms, but its head-specific proj…
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
Yufei Xu, Fanxu Meng, Fan Jiang +11
Token-level sparse attention mechanisms, exemplified by DeepSeek Sparse Attention (DSA), achieve fine-grained key selection by scoring every historical key for each query through a…
Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
Pingzhi Tang, Ruijie Zhou, Fanxu Meng +2
Current quantization methods for LLMs predominantly rely on block-wise structures to maintain efficiency, often at the cost of representational flexibility. In this work, we demons…
Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Models
Junru Lu, Jiarui Qin, Lingfeng Qiao +35
We introduce Youtu-LLM, a lightweight yet powerful language model that harmonizes high computational efficiency with native agentic intelligence. Unlike typical small models that r…