Normative Conflicts and Shallow AI Alignment

Abstract

The progress of AI systems such as large language models (LLMs) raises increasingly pressing concerns about their safe deployment. This paper examines the value alignment problem for LLMs, arguing that current alignment strategies are fundamentally inadequate to prevent misuse. Despite ongoing efforts to instill norms such as helpfulness, honesty, and harmlessness in LLMs through fine-tuning based on human preferences, they remain vulnerable to adversarial attacks that exploit conflicts between these norms. I argue that this vulnerability reflects a fundamental limitation of existing alignment methods: they reinforce shallow behavioral dispositions rather than endowing LLMs with a genuine capacity for normative deliberation. Drawing from on research in moral psychology, I show how humans' ability to engage in deliberative reasoning enhances their resilience against similar adversarial tactics. LLMs, by contrast, lack a robust capacity to detect and rationally resolve normative conflicts, leaving them susceptible to manipulation; even recent advances in reasoning-focused LLMs have not addressed this vulnerability. This ``shallow alignment'' problem carries significant implications for AI safety and regulation, suggesting that current approaches are insufficient for mitigating potential harms posed by increasingly capable AI systems.

arXiv commentPublished in Philosophical Studies

References in corpus (39)

• Survey of Hallucination in Natural Language Generation
• Training language models to follow instructions with human feedback
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Scaling Laws for Neural Language Models
• Evaluating Large Language Models Trained on Code
• Emergent Abilities of Large Language Models
• Artificial Intelligence, Values and Alignment
• Deep reinforcement learning from human preferences
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Adversarial Examples for Evaluating Reading Comprehension Systems
• Large Language Models Encode Clinical Knowledge
• Predictability and Surprise in Large Generative Models
• Quantifying Memorization Across Neural Language Models
• Improving alignment of dialogue agents via targeted human judgements
• Towards Understanding Sycophancy in Language Models
• Large Language Models for Software Engineering: A Systematic Literature Review
• Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study
• Large Language Models Struggle to Learn Long-Tail Knowledge
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
• OpenAI o1 System Card
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
• Measuring Faithfulness in Chain-of-Thought Reasoning
• A Philosophical Introduction to Language Models -- Part I: Continuity With Classic Debates
• Generating Phishing Attacks using ChatGPT
• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
• Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs
• Understanding the Effects of RLHF on LLM Generalisation and Diversity
• Deliberative Alignment: Reasoning Enables Safer Language Models
• Large Language Models for Scientific Synthesis, Inference and Explanation
• Will releasing the weights of future large language models grant widespread access to pandemic agents?
• Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models
• Passive learning of active causal strategies in agents and language models
• Structured Chemistry Reasoning with Large Language Models
• Improving Alignment and Robustness with Circuit Breakers
• EasyJailbreak: A Unified Framework for Jailbreaking Large Language Models
• Image Hijacks: Adversarial Images can Control Generative Models at Runtime
• Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks
• Safety Alignment Should Be Made More Than Just a Few Tokens Deep
• The Pitfalls of "Security by Obscurity" And What They Mean for Transparent AI