238 citations · 506 across the 29 of their papers we have counts for
33 papers
Spatial Induction Heads: In-Context Learning of Multidimensional Cellular Automata
Kimia Kazemian, Menghan Xu, John Thickstun +1
Induction heads provide a mechanistic account of in-context learning in sequential data, but existing theory largely assumes that the context relevant to a prediction forms a conti…
Large Language Continuous Diffusion Models
Zhihan Yang, Wei Guo, Jean-Marie Lemercier +14
Despite the success of discrete diffusion language models (dLMs) for fast parallel decoding, their non-smooth, high-dimensional space hinders trajectory steering for reasoning and…
Unlocking Lossless Speedups in LLMs via Discrete Diffusion
Subham Sekhar Sahoo, Lingjie Chen, Khiem Pham +14
Large Language Models (LLMs) owe much of their success to next-token prediction (NTP), but their autoregressive (AR) structure requires slow, sequential token generation. To overco…
From Forensics to Ecosystems: Rethinking Watermarks for Generative AI Oversight
Daniel Susser, John Thickstun, Gili Vidan
The arrival of generative AI as a cheap, widely accessible commercial service, and the tidal wave of AI-generated synthetic content it has unleashed, have provoked deep epistemic a…
Fidelity-Diversity Metrics for Text
Amanda Wang, Tudor Manole, Florentina Bunea +1
As language modeling technology matures, there is an increasing research focus on the composition and curation of datasets used to train these models. For instance, practitioners c…
Music Transcription with (Almost) No Supervision
Saebyeol Shin, Chao Wan, Zhenzhen Liu +4
Competitive music transcription models require large amounts of paired audio-score data, which is scarce due to collection costs, alignment difficulty, and copyright restrictions.…