8 papers · 1 filter
QuasiMoTTo: Quasi-Monte Carlo Test-Time Scaling
Michael Y. Li, Anthony Zhan, Kanishk Gandhi +2
Scaling inference compute, by generating many parallel attempts per problem, is a costly but reliable lever for improving language model capabilities. By default these attempts are…
Simplified Sparse Attention via Gist Tokens
Yuzhen Mao, Michael Y. Li, Emily B. Fox
Sparse attention can reduce the cost of long-context inference, but most variants introduce new architectural components. We introduce Simplified Sparse Attention (SSA), a simpler…
Neural Garbage Collection: Learning to Forget while Learning to Reason
Michael Y. Li, Jubayer Ibn Hamid, Emily B. Fox +1
Chain-of-thought reasoning has driven striking advances in language model capability, yet every reasoning step grows the KV cache, creating a bottleneck to scaling this paradigm fu…
What Should Embeddings Embed? Autoregressive Models Represent Latent Generating Distributions
Liyi Zhang, Michael Y. Li, R. Thomas McCoy +3
Autoregressive language models have demonstrated a remarkable ability to extract latent structure from text. The embeddings from large language models have been shown to capture as…
BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery
Kanishk Gandhi, Michael Y. Li, Lyle Goodyear +5
Understanding the world and explaining it with scientific theories is a central aspiration of artificial intelligence research. Proposing theories, designing experiments to test th…
Automated Hypothesis Validation with Agentic Sequential Falsifications
Kexin Huang, Ying Jin, Ryan Li +3
Hypotheses are central to information acquisition, decision-making, and discovery. However, many real-world hypotheses are abstract, high-level statements that are difficult to val…