12 papers
HiSpec: Hierarchical Speculative Decoding for LLMs
Avinash Kumar, Sujay Sanghavi, Poulami Das
Speculative decoding accelerates LLM inference by using a smaller draft model to speculate tokens that a larger target model verifies. Verification is often the bottleneck (e.g. ve…
Token Time Continuous Diffusion for Language Modeling
Parikshit Bansal, Sujay Sanghavi
In this paper we introduce token time continuous diffusion (TTCD), a new diffusion language model which (a) operates in continuous space, deterministically mapping Gaussian noise t…
RARe: Retrieval Augmented Retrieval with In-Context Examples
Atula Tejaswi, Yoonsang Lee, Sujay Sanghavi +1
While in-context learning is well-studied with decoder-only language models (LLMs), its utility for encoder-only models remains underexplored. We study in-context learning for enco…
Enabling Approximate Joint Sampling in Diffusion LMs
Parikshit Bansal, Sujay Sanghavi
In autoregressive language models, each token is sampled by conditioning on all the past tokens; the overall string has thus been sampled from the correct underlying joint distribu…
Context-Free Synthetic Data Mitigates Forgetting
Parikshit Bansal, Sujay Sanghavi
Fine-tuning a language model often results in a degradation of its existing performance on other tasks, due to a shift in the model parameters; this phenomenon is often referred to…
Understanding Self-Supervised Learning via Gaussian Mixture Models
Parikshit Bansal, Ali Kavis, Sujay Sanghavi
Self-supervised learning attempts to learn representations from un-labeled data; it does so via a loss function that encourages the embedding of a point to be close to that of its…