22 citations · 113 across the 46 of their papers we have counts for
3 papers · 1 filter
Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding
WenHung Lee, Jian-Jia Chen, Xiaolin Lin +6
While speculative decoding improves inference throughput for multi-batch long-context Large Language Models (LLMs), its efficiency is often limited by a verification bottleneck whe…
CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation
Xiaolin Lin, Jingcun Wang, Olga Kondrateva +3
Recent advances in large language models (LLMs) have significantly boosted long-context processing. However, the increasing key-value (KV) cache size poses critical challenges to m…
Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression
Jingcun Wang, Yu-Guang Chen, Ing-Chao Lin +2
Large Language Models (LLMs) have achieved remarkable breakthroughs. However, the huge number of parameters in LLMs require significant amount of memory storage in inference, which…