6 citations · 18 across the 15 of their papers we have counts for
8 papers · 1 filter
Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding
Weiye Shi, Fanxu Meng, Muhan Zhang
Multi-head latent attention (MLA) is increasingly important for long-context LLM inference because compact latent states replace the growing key-value (KV) cache and reduce decodin…
Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
Pingzhi Tang, Ruijie Zhou, Fanxu Meng +2
Current quantization methods for LLMs predominantly rely on block-wise structures to maintain efficiency, often at the cost of representational flexibility. In this work, we demons…
LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades
Yanan Li, Fanxu Meng, Muhan Zhang +3
As Large Language Models (LLMs) are frequently updated, LoRA weights trained on earlier versions quickly become obsolete. The conventional practice of retraining LoRA weights from…
HD-PiSSA: High-Rank Distributed Orthogonal Adaptation
Yiding Wang, Fauxu Meng, Xuefeng Zhang +3
Existing parameter-efficient fine-tuning (PEFT) methods for large language models (LLMs), such as LoRA and PiSSA, constrain model updates to low-rank subspaces, limiting their expr…
TransMLA: Multi-Head Latent Attention Is All You Need
Fanxu Meng, Pingzhi Tang, Xiaojuan Tang +3
In this paper, we present TransMLA, a framework that seamlessly converts any GQA-based pre-trained model into an MLA-based model. Our approach enables direct compatibility with Dee…
RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?
Haotian Xu, Xing Wu, Weinong Wang +11
Can scaling transform reasoning? In this work, we explore the untapped potential of scaling Long Chain-of-Thought (Long-CoT) data to 1000k samples, pioneering the development of a…
CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning
Fanxu Meng, Pingzhi Tang, Fan jiang +1
Decoder-only models generate tokens autoregressively by caching key/value vectors, but as the cache grows, inference becomes memory-bound. To address this issue, we introduce CLOVE…
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
Fanxu Meng, Zhaohui Wang, Muhan Zhang
To parameter-efficiently fine-tune (PEFT) large language models (LLMs), the low-rank adaptation (LoRA) method approximates the model changes throug…