2 citations · 3 across the 9 of their papers we have counts for
12 papers
Understanding AI from the Inside: The State of Mechanistic Interpretability for AI Safety
Fazl Barez, Tung-Yu Wu, Hunar Batra +24
Neural networks are likely to be among the most consequential artefacts in human history. As these systems become more capable and autonomous, their potential to cause societal har…
Cross-Model In-Context Neurofeedback Reveals (Limited) Activation Control in LLMs
Davide Baldelli, Robyn Kozierok, Mirko Bronzi +1
Probes on activation vectors are widely used to detect misbehavior in language models, but their success relies on the assumption that models cannot control what their activations…
PyINE: A Framework for Scalable Elicitation and Oversight via Code Execution
Pierre-Luc St-Charles, Alessandro Palmas, Damiano Fornasiere +5
Reasoning models can remain capable of solving a task while still defaulting to cheaper but misleading shortcuts. This creates a central oversight problem: when a model gives an an…
Efficient Safety Alignment of Language Models via Latent Personality Traits
Mohamed Amine Merzouk, Nolan Smyth, Damiano Fornasiere +3
Current safety methods for large language models are known to be vulnerable to adversarial attacks, motivating research into robust alternatives. Latent Adversarial Training (LAT)…
Bayesian Symbolic Regression with Entropic Reinforcement Learning
Oussama Boussif, Mohammed Mahfoud, Younesse Kaddar +6
Symbolic regression is the problem of finding an algebraic expression describing a stochastic dependence of a target variable on a set of inputs. Unlike forms of regression that fi…
Safety from Honesty in a Disinterested AI Predictor
Yoshua Bengio, Oliver Richardson, Tomáš Gavenčiak +13
As AI systems become more capable, training procedures that optimize for downstream outcomes risk introducing implicit agency: goal-directed behavior that designers never specified…
How Much is Left? LLMs Linearly Encode Their Remaining Output Length
Mohamed Amine Merzouk, Dmitri Carpov, Mirko Bronzi +2
Large language models generate one token at a time, yet their responses show remarkably consistent length structure: step-by-step solutions converge in predictable token counts, re…
Language models recognize dropout and Gaussian noise applied to their activations
Damiano Fornasiere, Mirko Bronzi, Spencer Kitts +3
We provide evidence that language models can detect, localize and, to a degree, verbalize the difference between a-semantic perturbations applied to their activations. More precise…
Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?
Yoshua Bengio, Michael Cohen, Damiano Fornasiere +10
The leading AI companies are increasingly focused on building generalist AI agents -- systems that can autonomously plan, act, and pursue goals across almost all tasks that humans…
Trees and spectra of Heyting algebras
Damiano Fornasiere, Tommaso Moraschini
A poset is Esakia representable when it is isomorphic to the prime spectrum of a Heyting algebra. Notably, every Esakia representable poset is also the spectrum of a commutative ri…
Can a Bayesian Oracle Prevent Harm from an Agent?
Yoshua Bengio, Michael K. Cohen, Nikolay Malkin +4
Is there a way to design powerful AI systems based on machine learning methods that would satisfy probabilistic safety guarantees? With the long-term goal of obtaining a probabilis…
Intuitionistic Sahlqvist theory for deductive systems
Damiano Fornasiere, Tommaso Moraschini
Sahlqvist theory is extended to the fragments of the intuitionistic propositional calculus that include the conjunction connective. This allows us to introduce a Sahlqvist theory o…