15 papers
Defensive Sufficiency in a Stackelberg Model of AI Security
Subhabrata Majumdar, Rajlakshmi Chavan
Feedback from automated testing, human red teaming, and incident response can strengthen an AI system's defenses when discovered failures lead to effective repairs. We study when t…
Removing Information Content Does Not Certify Tamper Resistance in Open-Weight Models
Domenic Rosati, Alessa Carbo, Ali Dadsetan +5
Does removing harmful information make open-weight models resistant to fine-tuning attacks? We show that mutual information at release alone cannot universally certify slow recover…
No Unique Minimizer, No Problem: On the Consistency of Robust Neural Classifiers
Subhabrata Majumdar, Anand Deo, Partha Pratim Saha +1
Neural network classifiers trained by cross-entropy minimization are highly sensitive to label noise and adversarial contamination. While robust alternatives offer bounded influenc…
OTAP: Structure-Aware Optimal Transport for Evaluating Planning and Execution in Agent Trajectories
Babak Barazandeh, Subhabrata Majumdar, George Michailidis
Large language model agents solve tasks by generating trajectories that interleave planning, tool calls, and intermediate results. Current evaluation metrics reduce such a trajecto…
Distribution-Specific Curvature Control with Finite-Sample Guarantees for Open-Weight Safety
Domenic Rosati, Ali Dadsetan, Hong Huang +5
A short fine-tuning run can undo the safety guards of an open-weight model---retraining a refusal-trained assistant to aid weapons development or produce hate speech. Preventing su…
Localized LoRA-MoE: Block-wise Low-Rank Experts With Adaptive Routing
Babak Barazandeh, Subhabrata Majumdar, Vinay Prithyani +1
Large Language Models (LLMs) and high-dimensional perception networks increasingly rely on parameter-efficient fine-tuning (PEFT) to adapt to diverse operational contexts. However,…
Limits of Reliability and Scaling in Language Models
Subhabrata Majumdar
Large language models (LLMs) are trained and evaluated as though perfect reliability is achievable for any task given sufficient scale. We show that this assumption is information-…
PsychoPass: Geometric Profiling of Multi-Turn Adversarial LLM Conversations
Muberra Ozmen, Subhabrata Majumdar
Multi-turn jailbreak attacks on large language models (LLMs) reveal a mismatch in current guardrails: they operate on individual turns, while attacks unfold as trajectories across…
Next-Billion AI Index: The compass for AI utility and adoption in the global majority
Ambrish Rawat, Jessica He, Subhabrata Majumdar +6
Generative AI assessments remain dominated by frontier capability benchmarks that often fail to capture whether systems can be sustainably deployed, adapted, and trusted in locally…
Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability
Harsh Raj, Niranjan Orkat, Suvrorup Mukherjee +3
This paper establishes a rigorous measurement science for AI agent reliability, providing a foundational framework for quantifying consistency under semantically preserving perturb…
Limits of Convergence-Rate Control for Open-Weight Safety
Domenic Rosati, Xijie Zeng, Hong Huang +4
Open-weight foundation models can be fine-tuned for harmful purposes after release, yet no existing training resistance methods provide theoretical guarantees. Treating these inter…
Consistency in Language Models: Current Landscape, Challenges, and Future Directions
Jekaterina Novikova, Carol Anderson, Borhane Blili-Hamelin +2
The hallmark of effective language use lies in consistency: expressing similar meanings in similar contexts and avoiding contradictions. While human communication naturally demonst…
Improving Consistency in Large Language Models through Chain of Guidance
Harsh Raj, Vipul Gupta, Domenic Rosati +1
Consistency is a fundamental dimension of trustworthiness in Large Language Models (LLMs). For humans to be able to trust LLM-based applications, their outputs should be consistent…
Evaluating Defences against Unsafe Feedback in RLHF
Domenic Rosati, Giles Edkins, Harsh Raj +5
While there has been progress towards aligning Large Language Models (LLMs) with human values and ensuring safe behaviour at inference time, safety guards can easily be removed whe…
Representation Noising: A Defence Mechanism Against Harmful Finetuning
Domenic Rosati, Jan Wehner, Kai Williams +7
Releasing open-source large language models (LLMs) presents a dual-use risk since bad actors can easily fine-tune these models for harmful purposes. Even without the open release o…