4 papers
Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction
Yijun Liu, Yixuan Wang, Yuzhuang Xu +4
Large language models (LLMs) utilize key-value (KV) cache to store historical information during sequence processing. The size of KV cache grows linearly as the length of the seque…
Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decoding
Yixuan Wang, Yijun Liu, Shiyu ji +4
Large language models (LLMs) suffer from high inference latency due to the auto-regressive decoding process. Speculative decoding accelerates inference by generating multiple draft…
Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query
Yixuan Wang, Shiyu Ji, Yijun Liu +4
Large language models (LLMs) rely on key-value cache (KV cache) to accelerate decoding by reducing redundant computations. However, the KV cache memory usage grows substantially wi…
A Survey on Transformer Context Extension: Approaches and Evaluation
Yijun Liu, Jinzheng Yu, Yang Xu +2
Large language models (LLMs) based on Transformer have been widely applied in the filed of natural language processing (NLP), demonstrating strong performance, particularly in hand…