1 citations · 2 across the 18 of their papers we have counts for
5 papers · 2 filters
Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match
Jinze Li, Yixing Xu, Guanchen Li +6
Large language models (LLMs) achieve strong performance across diverse tasks but suffer from high inference latency due to their autoregressive generation. Speculative Decoding (SP…
SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruning
Huanxuan Liao, Yixing Xu, Shizhu He +6
Long-context inference in large language models (LLMs) is increasingly constrained by the KV cache bottleneck: memory usage grows linearly with sequence length, while attention com…
Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
Jinze Li, Yixing Xu, Haiduo Huang +4
Speculative decoding (SPD) aims to accelerate the auto-regressive token generation process of a target Large Language Model (LLM). Some approaches employ a draft model with multipl…
Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoE
Haiduo Huang, Fuwei Yang, Zhenhua Liu +7
Speculative decoding (SD) accelerates large language model inference by using a smaller draft model to predict multiple tokens, which are then verified in parallel by the larger ta…
MSWA: Refining Local Attention with Multi-ScaleWindow Attention
Yixing Xu, Shivank Nag, Dong Li +2
Transformer-based LLMs have achieved exceptional performance across a wide range of NLP tasks. However, the standard self-attention mechanism suffers from quadratic time complexity…