5 papers
Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decoding
Rahul Thomas, Teo Kitanovski, Micah Goldblum +1
Multi-path speculative decoding accelerates lossless sampling from a target model by using a cheaper draft model to generate a draft tree of tokens, and then applies a verification…
Greedy Multi-Path Block Verification for Faster Decoding in Speculative Sampling
Rahul Thomas, Arka Pal
The goal of -step speculative decoding is to accelerate autoregressive decoding of a target model by using a cheaper draft model to generate a candidate path of tokens. Base…
Global Resolution: Optimal Multi-Draft Speculative Sampling via Convex Minimization
Rahul Krishna Thomas, Arka Pal
Speculative sampling reduces the latency of autoregressive decoding for target model LLMs without sacrificing inference quality, by using a cheap draft model to suggest a candidate…
Cascade: Token-Sharded Private LLM Inference
Rahul Thomas, Louai Zahran, Erica Choi +3
As LLMs continue to increase in parameter size, the computational resources required to run them are available to fewer parties. Therefore, third-party inference services -- where…
An Attack to Break Permutation-Based Private Third-Party Inference Schemes for LLMs
Rahul Thomas, Louai Zahran, Erica Choi +3
Recent advances in Large Language Models (LLMs) have led to the widespread adoption of third-party inference services, raising critical privacy concerns. Existing methods of perfor…