7 citations · 11 across the 13 of their papers we have counts for
17 papers
PIG: Parallel Inconsistency Guidance for Accelerating Diffusion Language Models
Quentin Fruytier, Anton Xue, Sanjay Shakkottai +1
Diffusion language models accelerate generation by decoding multiple masked positions per forward pass. However, accuracy is known to drop when parallel decoding trajectories devia…
TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Models
Deqing Fu, Huangyuan Su, Rajat Sen +4
Tabular foundation models achieve strong zero-shot accuracy on structured data by pretraining on synthetic tables, but they ignore the column names, task descriptions, and auxiliar…
dQwen3.5: Hybrid-Attention Diffusion Language Models
Anton Xue, Litu Rout, Aditya Akella +3
Adapting a pretrained autoregressive (AR) model is a cost-efficient route to a diffusion language model (DLM). While nearly all such adaptations start from a full-attention transfo…
Token Time Continuous Diffusion for Language Modeling
Parikshit Bansal, Sujay Sanghavi
In this paper we introduce token time continuous diffusion (TTCD), a new diffusion language model which (a) operates in continuous space, deterministically mapping Gaussian noise t…
Sculpting the Latent Space: Disentanglement with WAE-MMD and Programmable Priors
Quentin Fruytier, Akshay Malhotra, Shahab Hamidi-Rad +3
Learning disentangled representations, where semantic features are independently captured in a latent space, is dominated by Variational Auto Encoders (VAE) and Adversarial Trainin…
HiSpec: Hierarchical Speculative Decoding for LLMs
Avinash Kumar, Sujay Sanghavi, Poulami Das
Speculative decoding accelerates LLM inference by using a smaller draft model to speculate tokens that a larger target model verifies. Verification is often the bottleneck (e.g. ve…
Enabling Approximate Joint Sampling in Diffusion LMs
Parikshit Bansal, Sujay Sanghavi
In autoregressive language models, each token is sampled by conditioning on all the past tokens; the overall string has thus been sampled from the correct underlying joint distribu…
SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs
Jiacheng Lin, Zhongruo Wang, Kun Qian +14
Supervised Fine-Tuning (SFT) on domain-specific datasets is a common approach to adapt Large Language Models (LLMs) to specialized tasks but is often believed to degrade their gene…
Context-Free Synthetic Data Mitigates Forgetting
Parikshit Bansal, Sujay Sanghavi
Fine-tuning a language model often results in a degradation of its existing performance on other tasks, due to a shift in the model parameters; this phenomenon is often referred to…
Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting
Sunny Sanyal, Hayden Prairie, Rudrajit Das +2
Fine-tuning a pre-trained model on a downstream task often degrades its original capabilities, a phenomenon known as "catastrophic forgetting". This is especially an issue when one…
Learning Mixtures of Experts with EM: A Mirror Descent Perspective
Quentin Fruytier, Aryan Mokhtari, Sujay Sanghavi
Classical Mixtures of Experts (MoE) are Machine Learning models that involve partitioning the input space, with a separate "expert" model trained on each partition. Recently, MoE-b…
Understanding Self-Supervised Learning via Gaussian Mixture Models
Parikshit Bansal, Ali Kavis, Sujay Sanghavi
Self-supervised learning attempts to learn representations from un-labeled data; it does so via a loss function that encourages the embedding of a point to be close to that of its…
RARe: Retrieval Augmented Retrieval with In-Context Examples
Atula Tejaswi, Yoonsang Lee, Sujay Sanghavi +1
While in-context learning is well-studied with decoder-only language models (LLMs), its utility for encoder-only models remains underexplored. We study in-context learning for enco…
DataComp-LM: In search of the next generation of training sets for language models
Jeffrey Li, Alex Fang, Georgios Smyrnis +56
We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardize…
Adaptive and Optimal Second-order Optimistic Methods for Minimax Optimization
Ruichen Jiang, Ali Kavis, Qiujiang Jin +2
We propose adaptive, line search-free second-order methods with optimal rate of convergence for solving convex-concave min-max problems. By means of an adaptive step size, our algo…
SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors
Vijay Lingam, Atula Tejaswi, Aditya Vavre +7
Popular parameter-efficient fine-tuning (PEFT) methods, such as LoRA and its variants, freeze pre-trained model weights \(W\) and inject learnable matrices \(ΔW\). These \(ΔW\) mat…
Time Weaver: A Conditional Time Series Generation Model
Sai Shankar Narasimhan, Shubhankar Agarwal, Oguzhan Akcin +2
Imagine generating a city's electricity demand pattern based on weather, the presence of an electric vehicle, and location, which could be used for capacity planning during a winte…