9 papers
Towards a Neural Debugger for Python
Maximilian Beck, Jonas Gehring, Jannik Kossen +1
Training large language models (LLMs) on Python execution traces grounds them in code execution and enables the line-by-line execution prediction of whole Python programs, effectiv…
Scaling Up Active Testing to Large Language Models
Gabrielle Berrada, Jannik Kossen, Freddie Bickford Smith +3
Active testing enables label-efficient evaluation of predictive models through careful data acquisition, but it can pose a significant computational cost. We identify cost-saving m…
Estimating the Hallucination Rate of Generative AI
Andrew Jesson, Nicolas Beltran-Velez, Quentin Chu +5
This paper presents a method for estimating the hallucination rate for in-context learning (ICL) with generative AI. In ICL, a conditional generative model (CGM) is prompted with a…
CWM: An Open-Weights LLM for Research on Code Generation with World Models
FAIR CodeGen team, Jade Copet, Quentin Carbonneaux +48
We release Code World Model (CWM), a 32-billion-parameter open-weights LLM, to advance research on code generation with world models. To improve code understanding beyond what can…
Rethinking Aleatoric and Epistemic Uncertainty
Freddie Bickford Smith, Jannik Kossen, Eleanor Trollope +3
The ideas of aleatoric and epistemic uncertainty are widely used to reason about the probabilistic predictions of machine-learning models. We identify incoherence in existing discu…
Reducing Large Language Model Safety Risks in Women's Health using Semantic Entropy
Jahan C. Penny-Dimri, Magdalena Bachmann, William R. Cooke +7
Large language models (LLMs) hold substantial promise for clinical decision support. However, their widespread adoption in medicine, particularly in healthcare, is hindered by thei…