6 citations · 13 across the 15 of their papers we have counts for
18 papers
LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention
Zhaohui Wang, Zhixin Pan, Fanxu Meng +1
Sparse attention reduces core-attention computation, but its indexers still incur repeated selection work and per-layer key-cache storage. Reusing selected indices across layers re…
DoSA: Dots Sparse Attention
Fanxu Meng, Taosong Fang
Sparse attention visits few keys, yet finding them can still require scoring every key with every indexer head. We introduce DoSA (Dots Sparse Attention), a lossless method for mak…
Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding
Weiye Shi, Fanxu Meng, Muhan Zhang
Multi-head latent attention (MLA) is increasingly important for long-context LLM inference because compact latent states replace the growing key-value (KV) cache and reduce decodin…
MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
Ruijie Zhou, Fanxu Meng, Yufei Xu +4
DeepSeek Sparse Attention (DSA) sets the state of the art for fine-grained inference-time sparse attention by introducing a learned token-wise indexer that scores every prefix toke…
GQLA: Group-Query Latent Attention for Hardware-Adaptive Decoding
Fanxu Meng, Zhixin Pan
Decoding efficiency depends on KV-cache traffic, computation, and parallelism. Multi-head Latent Attention (MLA) offers absorbed and head-expanded forms, but its head-specific proj…
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
Yufei Xu, Fanxu Meng, Fan Jiang +11
Token-level sparse attention mechanisms, exemplified by DeepSeek Sparse Attention (DSA), achieve fine-grained key selection by scoring every historical key for each query through a…
Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
Pingzhi Tang, Ruijie Zhou, Fanxu Meng +2
Current quantization methods for LLMs predominantly rely on block-wise structures to maintain efficiency, often at the cost of representational flexibility. In this work, we demons…
Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Models
Junru Lu, Jiarui Qin, Lingfeng Qiao +35
We introduce Youtu-LLM, a lightweight yet powerful language model that harmonizes high computational efficiency with native agentic intelligence. Unlike typical small models that r…
Law in Silico: Simulating Legal Society with LLM-Based Agents
Yiding Wang, Yuxuan Chen, Fanxu Meng +3
Since real-world legal experiments are often costly or infeasible, simulating legal societies with Artificial Intelligence (AI) systems provides an effective alternative for verify…
TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference
Xiaojuan Tang, Fanxu Meng, Pingzhi Tang +4
Multi-Head Latent Attention (MLA), introduced in DeepSeek-V2, compresses key-value states into a low-rank latent vector, caching only this vector to reduce memory. In tensor parall…
LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades
Yanan Li, Fanxu Meng, Muhan Zhang +3
As Large Language Models (LLMs) are frequently updated, LoRA weights trained on earlier versions quickly become obsolete. The conventional practice of retraining LoRA weights from…
LIFT: A Novel Framework for Enhancing Long-Context Understanding of LLMs via Long Input Fine-Tuning
Yansheng Mao, Yufei Xu, Jiaqi Li +5
Long-context understanding remains challenging for LLMs due to limited context windows. This paper introduces Long Input Fine-Tuning (LIFT), a framework that improves the long-cont…
TransMLA: Multi-Head Latent Attention Is All You Need
Fanxu Meng, Pingzhi Tang, Xiaojuan Tang +3
In this paper, we present TransMLA, a framework that seamlessly converts any GQA-based pre-trained model into an MLA-based model. Our approach enables direct compatibility with Dee…
LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning
Yansheng Mao, Jiaqi Li, Fanxu Meng +3
Long context understanding remains challenging for large language models due to their limited context windows. This paper introduces Long Input Fine-Tuning (LIFT) for long context…
CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning
Fanxu Meng, Pingzhi Tang, Fan jiang +1
Decoder-only models generate tokens autoregressively by caching key/value vectors, but as the cache grows, inference becomes memory-bound. To address this issue, we introduce CLOVE…
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
Fanxu Meng, Zhaohui Wang, Muhan Zhang
To parameter-efficiently fine-tune (PEFT) large language models (LLMs), the low-rank adaptation (LoRA) method approximates the model changes throug…
Chain of Images for Intuitively Reasoning
Fanxu Meng, Haotong Yang, Yiding Wang +1
The human brain is naturally equipped to comprehend and interpret visual information rapidly. When confronted with complex problems or concepts, we use flowcharts, sketches, and di…
Parrot Mind: Towards Explaining the Complex Task Reasoning of Pretrained Large Language Models with Template-Content Structure
Haotong Yang, Fanxu Meng, Zhouchen Lin +1
The pre-trained large language models (LLMs) have shown their extraordinary capacity to solve reasoning tasks, even on tasks that require a complex process involving multiple sub-s…