54 citations · 87 across the 20 of their papers we have counts for
17 papers · 1 filter
Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models
Guanghan Wang, Bing He, Zhan Shi +3
Diffusion language models (DLMs) have attracted wide attention for their fast, parallel decoding, but they are believed to be far less efficient to train than autoregressive langua…
Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Models
Yair Schiff, Omer Belhasin, Roy Uziel +7
Recent works on continuous diffusion for discrete data have demonstrated performance on par with comparable discrete diffusion models. However, these continuous counterparts lack k…
Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding
Marianne Arriola, Volodymyr Kuleshov
Discrete diffusion models have steadily improved in quality relative to autoregressive (AR) models. However, these models are normally constrained to fixed-length generation and do…
DUEL: Exact Likelihood for Masked Diffusion via Deterministic Unmasking
Gilad Turok, Chris De Sa, Volodymyr Kuleshov
Masked diffusion models (MDMs) generate text by iteratively selecting positions to unmask and then predicting tokens at those positions. Yet MDMs lack proper likelihood evaluation:…
Learn from Your Mistakes: Self-Correcting Masked Diffusion Models
Yair Schiff, Omer Belhasin, Roy Uziel +6
Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models, enabling parallel token generation while achieving competitive performance. Despite…
Encoder-Decoder Diffusion Language Models for Efficient Training and Inference
Marianne Arriola, Yair Schiff, Hao Phung +2
Discrete diffusion models enable parallel token sampling for faster inference than autoregressive approaches. However, prior diffusion models use a decoder-only architecture, which…
d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation
Guanghan Wang, Gilad Turok, Yair Schiff +2
While diffusion language models (DLMs) have achieved competitive performance in text generation, improving their reasoning ability with reinforcement learning remains an active res…
The Diffusion Duality
Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan +3
Uniform-state discrete diffusion models hold the promise of fast text generation due to their inherent ability to self-correct. However, they are typically outperformed by autoregr…
RanDeS: Randomized Delta Superposition for Multi-Model Compression
Hangyu Zhou, Aaron Gokaslan, Volodymyr Kuleshov +1
From a multi-model compression perspective, model merging enables memory-efficient serving of multiple models fine-tuned from the same base, but suffers from degraded performance d…
Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
Marianne Arriola, Aaron Gokaslan, Justin T. Chiu +5
Diffusion language models offer unique benefits over autoregressive models due to their potential for parallelized generation and controllability, yet they lag in likelihood modeli…
Remasking Discrete Diffusion Models with Inference-Time Scaling
Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo +1
Part of the success of diffusion models stems from their ability to perform iterative refinement, i.e., repeatedly correcting outputs during generation. However, modern masked disc…
The GAN is dead; long live the GAN! A Modern GAN Baseline
Yiwen Huang, Aaron Gokaslan, Volodymyr Kuleshov +1
There is a widely-spread claim that GANs are difficult to train, and GAN architectures in the literature are littered with empirical tricks. We provide evidence against this claim…
Simple Guidance Mechanisms for Discrete Diffusion Models
Yair Schiff, Subham Sekhar Sahoo, Hao Phung +7
Diffusion models for continuous data gained widespread adoption owing to their high quality generation and control mechanisms. However, controllable diffusion on discrete data face…
DySLIM: Dynamics Stable Learning by Invariant Measure for Chaotic Systems
Yair Schiff, Zhong Yi Wan, Jeffrey B. Parker +4
Learning dynamics from dissipative chaotic systems is notoriously difficult due to their inherent instability, as formalized by their positive Lyapunov exponents, which exponential…
QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
Albert Tseng, Jerry Chee, Qingyao Sun +2
Post-training quantization (PTQ) reduces the memory footprint of LLMs by quantizing their weights to low-precision. In this work, we introduce QuIP#, a weight-only PTQ method that…
Diffusion Models With Learned Adaptive Noise
Subham Sekhar Sahoo, Aaron Gokaslan, Chris De Sa +1
Diffusion models have gained traction as powerful algorithms for synthesizing high-quality images. Central to these algorithms is the diffusion process, a set of equations which ma…
ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers
Junjie Yin, Jiahao Dong, Yingheng Wang +2
We propose a memory-efficient finetuning algorithm for large language models (LLMs) that supports finetuning LLMs with 65B parameters in 2/3/4-bit precision on as little as one 24G…