29 citations · 36 across the 6 of their papers we have counts for
7 papers
Conformal Language Modeling via Posterior Sampling
Nicolas Emmenegger, Theo X. Olausson, Armando Solar-Lezama +1
Large Language Models remain plagued by hallucinations. Recent work has sought to tame their prevalence using statistical techniques based on conformal prediction, with both theore…
An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models
Mingzhong Sun, Teresa Yeo, Armando Solar-Lezama +1
Studies of human reasoning have shown that people are typically stronger at evaluating reasoning than producing it from scratch. In contrast, large reasoning models (LRMs) are trai…
A Tale of Two Temperatures: Simple, Efficient, and Diverse Sampling from Diffusion Language Models
Theo X. Olausson, Metod Jazbec, Xi Wang +4
Much work has been done on designing fast and accurate sampling for diffusion language models (dLLMs). However, these efforts have largely focused on the tradeoff between speed and…
Challenges and Paths Towards AI for Software Engineering
Alex Gu, Naman Jain, Wen-Ding Li +7
AI for software engineering has made remarkable progress recently, becoming a notable success within generative AI. Despite this, there are still many challenges that need to be ad…
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
Naman Jain, King Han, Alex Gu +7
Large Language Models (LLMs) applied to code-related applications have emerged as a prominent field, attracting significant interest from both academia and industry. However, as ne…
The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?
Alex Gu, Wen-Ding Li, Naman Jain +4
While language models are increasingly more proficient at code generation, they still frequently generate incorrect programs. Many of these programs are obviously wrong, but others…
CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
Alex Gu, Baptiste Rozière, Hugh Leather +3
We present CRUXEval (Code Reasoning, Understanding, and eXecution Evaluation), a benchmark consisting of 800 Python functions (3-13 lines). Each function comes with an input-output…