44 citations · 145 across the 23 of their papers we have counts for
24 papers
From Constitutions to Control: Interpretable Rewards for Aligning Language Models
Johann D. Gaebler, Calvin Isley, Max Lamparth +2
Current approaches to aligning language models often make it hard to know what behavior is being rewarded or to change that reward in a targeted way. In particular, standard prefer…
Receptiveness, Not Sycophancy: Distinguishing Engagement from Deference in Language Models
Calvin Isley, Johann Gaebler, Max Lamparth +2
A central concern with language models is sycophancy: their tendency to defer to users' views at the expense of independent substantive judgment. In parallel, work on social sycoph…
Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting
Avijit Ghosh, Anka Reuel, Jenny Chim +45
AI evaluation results are produced at scale but reported inconsistently across leaderboards, model cards, benchmark papers, and company blogs. The cost is interpretive: readers can…
Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure
Max Lamparth, Daniel Fein, Andreas Haupt +2
Single-axis mitigations of reward-model biases (e.g., reducing proxy reliance on length, sycophancy, or style) can rotate optimization pressure onto correlated proxies rather than…
One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
Daniel Fein, Max Lamparth, Violet Xiang +2
Reward Models (RMs) are crucial for online alignment of language models (LMs) with human preferences. However, RM-based preference-tuning is vulnerable to reward hacking, whereby L…
Expert Evaluation and the Limits of Human Feedback in Mental Health AI Safety Testing
Kiana Jafari, Paul Ulrich Nikolaus Rust, Duncan Eddy +7
Learning from human feedback~(LHF) assumes that expert judgments, appropriately aggregated, yield valid ground truth for training and evaluating AI systems. We tested this assumpti…
Measuring and mitigating overreliance to build human-compatible AI
Lujain Ibrahim, Katherine M. Collins, Sunnie S. Y. Kim +14
Large language models (LLMs) distinguish themselves from previous technologies by functioning as collaborative ``thought partners,'' capable of engaging more fluidly in natural lan…
Multi-Agent Risks from Advanced AI
Lewis Hammond, Alan Chan, Jesse Clifton +41
The rapid development of advanced AI agents and the imminent deployment of many instances of these agents will give rise to multi-agent systems of unprecedented complexity. These s…
Moving Beyond Medical Exams: A Clinician-Annotated Fairness Dataset of Real-World Tasks and Ambiguity in Mental Healthcare
Max Lamparth, Declan Grabb, Amy Franks +8
Current medical language model (LM) benchmarks often over-simplify the complexities of day-to-day clinical practice tasks and instead rely on evaluating LMs on multiple-choice boar…
On the Trustworthiness of Generative Foundation Models: Guideline, Assessment, and Perspective
Yue Huang, Chujie Gao, Siyuan Wu +63
Generative Foundation Models (GenFMs) have emerged as transformative tools. However, their widespread adoption raises critical concerns regarding trustworthiness across dimensions.…
Prioritization First, Principles Second: An Adaptive Interpretation of Helpful, Honest, and Harmless Principles
Yue Huang, Chujie Gao, Yujun Zhou +5
The Helpful, Honest, and Harmless (HHH) principle is a foundational framework for aligning AI systems with human values. However, existing interpretations of the HHH principle ofte…
Humanity's Last Exam
Long Phan, Alice Gatti, Ziwen Han +1168
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achi…
BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices
Anka Reuel, Amelia Hardy, Chandler Smith +3
AI models are increasingly prevalent in high-stakes environments, necessitating thorough assessment of their capabilities and risks. Benchmarks are popular for measuring these attr…
Measuring Free-Form Decision-Making Inconsistency of Language Models in Military Crisis Simulations
Aryan Shrivastava, Jessica Hullman, Max Lamparth
There is an increasing interest in using language models (LMs) for automated decision-making, with multiple countries actively testing LMs to aid in military crisis decision-making…
Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation
Declan Grabb, Max Lamparth, Nina Vasan
Amidst the growing interest in developing task-autonomous AI for automated mental health care, this paper addresses the ethical and practical challenges associated with the issue a…
Markovian Transformers for Informative Language Modeling
Scott Viteri, Max Lamparth, Peter Chatain +1
Chain-of-Thought (CoT) reasoning often fails to faithfully reflect a language model's underlying decision process. We address this by introducing a Markovian language model framewo…
Human vs. Machine: Behavioral Differences Between Expert Humans and Language Models in Wargame Simulations
Max Lamparth, Anthony Corso, Jacob Ganz +3
To some, the advent of artificial intelligence (AI) promises better decision-making and increased military effectiveness while reducing the influence of human error and emotions. H…
Escalation Risks from Language Models in Military and Diplomatic Decision-Making
Juan-Pablo Rivera, Gabriel Mukobi, Anka Reuel +3
Governments are increasingly considering integrating autonomous AI agents in high-stakes military and foreign-policy decision-making, especially with the emergence of advanced gene…
TomOpt: Differential optimisation for task- and constraint-aware design of particle detectors in the context of muon tomography
Giles C. Strong, Maxime Lagrange, Aitor Orio +11
We describe a software package, TomOpt, developed to optimise the geometrical layout and specifications of detectors designed for tomography by scattering of cosmic-ray muons. The…
Analyzing And Editing Inner Mechanisms Of Backdoored Language Models
Max Lamparth, Anka Reuel
Poisoning of data sets is a potential security threat to large language models that can lead to backdoored models. A description of the internal mechanisms of backdoored language m…
Virgo: Scalable Unsupervised Classification of Cosmological Shock Waves
Max Lamparth, Ludwig Böss, Ulrich Steinwandel +1
Cosmological shock waves are essential to understanding the formation of cosmological structures. To study them, scientists run computationally expensive high-resolution 3D hydrody…
Gaussian Processes and Bayesian Optimization for High Precision Experiments
Max Lamparth, Mattis Bestehorn, Bastian Märkisch
High-precision measurements require optimal setups and analysis tools to achieve continuous improvements. Systematic corrections need to be modeled with high accuracy and known unc…
Toward the End-to-End Optimization of Particle Physics Instruments with Differentiable Programming: a White Paper
Tommaso Dorigo, Andrea Giammanco, Pietro Vischia +33
The full optimization of the design and operation of instruments whose functioning relies on the interaction of radiation with matter is a super-human task, given the large dimensi…
Design of the Magnet System of the Neutron Decay Facility PERC
Xiangzun Wang, Carmen Ziener, Hartmut Abele +27
The PERC (Proton and Electron Radiation Channel) facility is currently under construction at the research reactor FRM II, Garching. It will serve as an intense and clean source of…