112 citations · 228 across the 11 of their papers we have counts for
11 papers
Factored Verification: Detecting and Reducing Hallucination in Summaries of Academic Papers
Charlie George, Andreas Stuhlmüller
Hallucination plagues even frontier LLMs--but how bad is it really for summarizing academic papers? We evaluate Factored Verification, a simple automated method for detecting hallu…
Iterated Decomposition: Improving Science Q&A by Supervising Reasoning Processes
Justin Reppert, Ben Rachbach, Charlie George +4
Language models (LMs) can perform complex reasoning either end-to-end, with hidden latent state, or compositionally, with transparent intermediate state. Composition offers benefit…
RAFT: A Real-World Few-Shot Text Classification Benchmark
Neel Alex, Eli Lifland, Lewis Tunstall +9
Large pre-trained language models have shown promise for few-shot learning, completing text-based tasks given only a few task-specific examples. Will models soon solve classificati…
Evaluating Compositionality in Sentence Embeddings
Ishita Dasgupta, Demi Guo, Andreas Stuhlmüller +2
An important challenge for human-like AI is compositional semantics. Recent research has attempted to address this by using deep neural networks to learn vector space embeddings of…
Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
William Saunders, Girish Sastry, Andreas Stuhlmueller +1
AI systems are increasingly applied to complex tasks that involve interaction with humans. During training, such systems are potentially dangerous, as they haven't yet learned to a…
Agent-Agnostic Human-in-the-Loop Reinforcement Learning
David Abel, John Salvatier, Andreas Stuhlmüller +1
Providing Reinforcement Learning agents with expert advice can dramatically improve various aspects of learning. Prior work has developed teaching protocols that enable agents to l…
Learning the Preferences of Ignorant, Inconsistent Agents
Owain Evans, Andreas Stuhlmueller, Noah D. Goodman
An important use of machine learning is to learn what people value. What posts or photos should a user be shown? Which jobs or activities would a person find rewarding? In each cas…
Coarse-to-Fine Sequential Monte Carlo for Probabilistic Programs
Andreas Stuhlmüller, Robert X. D. Hawkins, N. Siddharth +1
Many practical techniques for probabilistic inference require a sequence of distributions that interpolate between a tractable distribution and an intractable distribution of inter…
C3: Lightweight Incrementalized MCMC for Probabilistic Programs using Continuations and Callsite Caching
Daniel Ritchie, Andreas Stuhlmüller, Noah D. Goodman
Lightweight, source-to-source transformation approaches to implementing MCMC for probabilistic programming languages are popular for their simplicity, support of existing determini…
A Dynamic Programming Algorithm for Inference in Recursive Probabilistic Programs
Andreas Stuhlmüller, Noah D. Goodman
We describe a dynamic programming algorithm for computing the marginal distribution of discrete probabilistic programs. This algorithm takes a functional interpreter for an arbitra…
Inducing Probabilistic Programs by Bayesian Program Merging
Irvin Hwang, Andreas Stuhlmüller, Noah D. Goodman
This report outlines an approach to learning generative models from data. We express models as probabilistic programs, which allows us to capture abstract patterns within the examp…