5 papers
Tsubame: Tree Replay for Diffusion-Based Speculative Decoding
Yepeng Weng, Qiao Hu, Takehisa Yairi
Context-aware dynamic trees allocate the speculative decoding budget according to draft path probabilities, adapting their depth and branching to the current context. Under stochas…
RheoSampling: Resolving the One-Hot Dilemma in Stochastic Dynamic-Tree Speculative Decoding
Qiao Hu, Yepeng Weng, Bo Zhang +1
Speculative decoding accelerates LLM inference by drafting multiple tokens in parallel, with tree-based methods further improving efficiency through hierarchical structures. Dynami…
UniVer: A Unified Perspective for Multi-step and Multi-draft Speculative Decoding
Yepeng Weng, Qiao Hu, Takehisa Yairi
Speculative decoding accelerates Large Language Models via draft-then-verify, where verification can be framed as an Optimal Transport (OT) problem. Existing approaches typically h…
Traversal Verification for Speculative Tree Decoding
Yepeng Weng, Qiao Hu, Xujie Chen +5
Speculative decoding is a promising approach for accelerating large language models. The primary idea is to use a lightweight draft model to speculate the output of the target mode…
Hierarchical graph sampling based minibatch learning with chain preservation and variance reduction
Qia Hu, Bo Jiao
Graph sampling-based Graph Convolutional Networks (GCNs) decouple sampling from forward and backward propagation during minibatch training, enhancing scalability with respect to la…