29 citations · 54 across the 9 of their papers we have counts for
5 papers · 1 filter
The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation
Lawrence Stewart, Matthew Trager, Sujan Kumar Gonugondla +1
Speculative decoding aims to speed up autoregressive generation of a language model by verifying in parallel the tokens generated by a smaller draft model.In this work, we explore…
Approximately Aligned Decoding
Daniel Melcer, Sujan Gonugondla, Pramuditha Perera +7
It is common to reject undesired outputs of Large Language Models (LLMs); however, current methods to do so require an excessive amount of computation to re-sample after a rejectio…
BASS: Batched Attention-optimized Speculative Sampling
Haifeng Qian, Sujan Kumar Gonugondla, Sungsoo Ha +6
Speculative decoding has emerged as a powerful method to improve latency and throughput in hosting large language models. However, most existing implementations focus on generating…
Token Alignment via Character Matching for Subword Completion
Ben Athiwaratkun, Shiqi Wang, Mingyue Shang +7
Generative models, widely utilized in various applications, can often struggle with prompts corresponding to partial tokens. This struggle stems from tokenization, where partial to…
Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs
Ben Athiwaratkun, Sujan Kumar Gonugondla, Sanjay Krishna Gouda +10
This study introduces bifurcated attention, a method designed to enhance language model inference in shared-context batch decoding scenarios. Our approach addresses the challenge o…