157 citations · 529 across the 78 of their papers we have counts for
84 papers
CAT-alyzing Honesty: Continuous Adversarial Training for Transferable LLM Honesty
Yuan-Chen Chang, Niklas Herbster, Gauthier Gidel +1
Large language models (LLMs) are becoming prevalent in scenarios where honesty is critical. However, despite alignment training, these models still exhibit dishonest behaviour, suc…
On the Stability and Convergence of Flow Map Training
Ishan Nath, Tom Stanic, Avishek Joey Bose +2
Flow maps are a recent method of probabilistic generation that amortize diffusion trajectories into one-step samplers, unlocking efficient generation. While progress has been made…
Visual Memory Attacks Can Persist Through The KV Cache
David Dobre, Leo Schwinn, Gauthier Gidel +3
Modern language model systems operate autonomously over increasingly long contexts containing untrusted text and images. Can an adversarial input continue to steer a model even aft…
Quantifying Overclaiming Propensity in Frontier LLM Agents
Nolan Smyth, Yorguin-Jose Mantilla-Ramos, Pascal Jr Tikeng Notsawo +6
Frontier coding agents are increasingly trusted to work autonomously for long periods of time, yet what they actually did is often hard to tell from their final response. We quanti…
Accelerated and Stable Convergence with Anchored Generalized Optimistic Method
Motahareh Sohrabi, Jianxin You, Simon Lacoste-Julien +2
We study first-order methods for solving monotone variational inequalities arising in min-max optimization. Classical approaches such as the extragradient method rely on two gradie…
Why Open Source? A Game-Theoretic Analysis of the AI Race
Andjela Mladenovic, Aaron Courville, Gauthier Gidel
In recent years, with the advancement of frontier AI, we have observed certain dynamics in open-sourcing and closed-sourcing decisions. We propose a game-theoretic model to analyze…
Activation Steering for Aligned Open-ended Generation without Sacrificing Coherence
Niklas Herbster, Martin Zborowski, Alberto Tosato +2
Alignment in LLMs is more brittle than commonly assumed: misalignment can be induced by adversarial prompts, benign fine-tuning, emergent misalignment, and goal misgeneralization.…
A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness
Leo Schwinn, Moritz Ladenburger, Tim Beyer +3
Automated \enquote{LLM-as-a-Judge} frameworks have become the de facto standard for scalable evaluation across natural language processing. For instance, in safety evaluation, thes…
Logarithmic-time Schedules for Scaling Language Models with Momentum
Damien Ferbach, Courtney Paquette, Gauthier Gidel +2
In practice, the hyperparameters and weight-decay in AdamW are typically kept at fixed values. Is there any reason to do otherwise? We show that for large-scale la…
Discrete Compositional Generation via General Soft Operators and Robust Reinforcement Learning
Marco Jiralerspong, Esther Derman, Danilo Vucetic +5
A major bottleneck in scientific discovery consists of narrowing an exponentially large set of objects, such as proteins or molecules, to a small set of promising candidates with d…
Jailbreak Distillation: Renewable Safety Benchmarking
Jingyu Zhang, Ahmed Elgohary, Xiawei Wang +5
Large language models (LLMs) are rapidly deployed in critical applications, raising urgent needs for robust safety benchmarking. We propose Jailbreak Distillation (JBDistill), a no…
Dimension-adapted Momentum Outscales SGD
Damien Ferbach, Katie Everett, Gauthier Gidel +2
We investigate scaling laws for stochastic momentum algorithms with small batch on the power law random features model, parameterized by data complexity, target complexity, and mod…
LLM-Safety Evaluations Lack Robustness
Tim Beyer, Sophie Xhonneux, Simon Geisler +3
In this paper, we argue that current safety alignment research efforts for large language models are hindered by many intertwined sources of noise, such as small datasets, methodol…
Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectives
Leo Schwinn, Yan Scholten, Tom Wollschläger +4
Misaligned research objectives have considerably hindered progress in adversarial robustness research over the past decade. For instance, an extensive focus on optimizing target me…
A Generative Approach to LLM Harmfulness Mitigation with Red Flag Tokens
David Dobre, Mehrnaz Mofakhami, Sophie Xhonneux +2
Many safety post-training methods for large language models (LLMs) are designed to modify the model's behaviour from producing unsafe answers to issuing refusals. However, such dis…
Tight Lower Bounds and Improved Convergence in Performative Prediction
Pedram Khorsandi, Rushil Gupta, Mehrnaz Mofakhami +2
Performative prediction is a framework accounting for the shift in the data distribution induced by the prediction of a model deployed in the real world. Ensuring rapid convergence…
Solving Hidden Monotone Variational Inequalities with Surrogate Losses
Ryan D'Orazio, Danilo Vucetic, Zichu Liu +3
Deep learning has proven to be effective in a wide variety of loss minimization problems. However, many applications of interest, like minimizing projected Bellman error and min-ma…
Investigating the Benefits of Nonlinear Action Maps in Data-Driven Teleoperation
Michael Przystupa, Gauthier Gidel, Matthew E. Taylor +3
As robots become more common for both able-bodied individuals and those living with a disability, it is increasingly important that lay people be able to drive multi-degree-of-free…
General Causal Imputation via Synthetic Interventions
Marco Jiralerspong, Thomas Jiralerspong, Vedant Shah +2
Given two sets of elements (such as cell types and drug compounds), researchers typically only have access to a limited subset of their interactions. The task of causal imputation…
Performative Prediction on Games and Mechanism Design
António Góis, Mehrnaz Mofakhami, Fernando P. Santos +2
Agents often have individual goals which depend on a group's actions. If agents trust a forecast of collective action and adapt strategically, such prediction can influence outcome…
Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences
Damien Ferbach, Quentin Bertrand, Avishek Joey Bose +1
The rapid progress in generative models has resulted in impressive leaps in generation quality, blurring the lines between synthetic and real data. Web-scale datasets are now prone…
Stochastic Frank-Wolfe: Unified Analysis and Zoo of Special Cases
Ruslan Nazykov, Aleksandr Shestakov, Vladimir Solodkin +3
The Conditional Gradient (or Frank-Wolfe) method is one of the most well-known methods for solving constrained optimization problems appearing in various machine learning tasks. Th…
Learning diverse attacks on large language models for robust red-teaming and safety tuning
Seanie Lee, Minsu Kim, Lynn Cherif +8
Red-teaming, or identifying prompts that elicit harmful responses, is a critical step in ensuring the safe and responsible deployment of large language models (LLMs). Developing ef…
Efficient Adversarial Training in LLMs with Continuous Attacks
Sophie Xhonneux, Alessandro Sordoni, Stephan Günnemann +2
Large language models (LLMs) are vulnerable to adversarial attacks that can bypass their safety guardrails. In many domains, adversarial training has proven to be one of the most p…