8 papers
Privacy-Preserving Mechanisms Enable Cheap Verifiable Inference of LLMs
Arka Pal, Louai Zahran, William Gvozdjak +2
As large language models (LLMs) continue to grow in size, fewer users are able to host and run models locally. This has led to increased use of third-party hosting services. Howeve…
Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decoding
Rahul Thomas, Teo Kitanovski, Micah Goldblum +1
Multi-path speculative decoding accelerates lossless sampling from a target model by using a cheaper draft model to generate a draft tree of tokens, and then applies a verification…
Greedy Multi-Path Block Verification for Faster Decoding in Speculative Sampling
Rahul Thomas, Arka Pal
The goal of -step speculative decoding is to accelerate autoregressive decoding of a target model by using a cheaper draft model to generate a candidate path of tokens. Base…
Knowing What You Know Is Not Enough: Large Language Model Confidences Don't Align With Their Actions
Arka Pal, Teo Kitanovski, Arthur Liang +2
Large language models (LLMs) are increasingly deployed in agentic and multi-turn workflows where they are tasked to perform actions of significant consequence. In order to deploy t…
Large Language Models Must Be Taught to Know What They Don't Know
Sanyam Kapoor, Nate Gruver, Manley Roberts +7
When using large language models (LLMs) in high-stakes applications, we need to know when we can trust their predictions. Some works argue that prompting high-performance LLMs is s…
Cascade: Token-Sharded Private LLM Inference
Rahul Thomas, Louai Zahran, Erica Choi +3
As LLMs continue to increase in parameter size, the computational resources required to run them are available to fewer parties. Therefore, third-party inference services -- where…