4 papers
PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Length
Situo Zhang, Yifan Zhang, Zichen Zhu +5
Speculative decoding (SD) is a powerful technique for accelerating the inference process of large language models (LLMs) without sacrificing accuracy. Typically, SD employs a small…
ProgRM: Build Better GUI Agents with Progress Rewards
Danyang Zhang, Situo Zhang, Ziyue Yang +5
LLM-based (Large Language Model) GUI (Graphical User Interface) agents can potentially reshape our daily lives significantly. However, current LLM-based GUI agents suffer from the…
AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures
Situo Zhang, Hankun Wang, Da Ma +4
Speculative Decoding (SD) is a popular lossless technique for accelerating the inference of Large Language Models (LLMs). We show that the decoding speed of SD frameworks with stat…
Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
Da Ma, Lu Chen, Situo Zhang +8
The rapid expansion of context window sizes in Large Language Models~(LLMs) has enabled them to tackle increasingly complex tasks involving lengthy documents. However, this progres…