4 papers
ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents
Yichu Fang, Sitong Wei, Haozhe Hu +1
Agentic language models repeatedly encode tool and skill schemas that recur across requests in different combinations and orders, preventing standard prefix caching from reusing th…
WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning
Haozhe Hu, Hao Wu, Peiran Yin +3
Pruning is a promising approach for improving the efficiency of LLMs. Existing static structured pruning methods are hardware-friendly and can deliver practical throughput gains, b…
Beyond Single-Dimensional Compression: The Compound Sparsity Frontier of Large Language Models
Chao Han, Haozhe Hu, Xiaoyu Shen
Large language models (LLMs) are often compressed through static parameter pruning or dynamic token-level computation, yet aggressive sparsification can trigger rapid performance d…
UniRank: Unified Rank Allocation for Low-Rank LLM Compression
Chao Han, Haozhe Hu, Fei Ma +2
Low-rank decomposition serves as a promising compression paradigm for large language models, however, rank allocation remains challenging: manual rules lack generalizability, and l…