Constitutional AI: Harmlessness from AI Feedback

Abstract

As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use 'RL from AI Feedback' (RLAIF). As a result we are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them. Both the SL and RL methods can leverage chain-of-thought style reasoning to improve the human-judged performance and transparency of AI decision making. These methods make it possible to control AI behavior more precisely and with far fewer human labels.

Cited by in corpus (28)

• Machine Culture
• A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
• UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility
• Exploring the psychology of LLMs' Moral and Legal Reasoning
• Identifying and Mitigating the Security Risks of Generative AI
• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• Escalation Risks from Language Models in Military and Diplomatic Decision-Making
• The current status of large language models in summarizing radiology report impressions
• Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties
• REFINER: Reasoning Feedback on Intermediate Representations
• From Paper to Card: Transforming Design Implications with Generative AI
• Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Models
• AI-induced sexual harassment: Investigating Contextual Characteristics and User Reactions of Sexual Harassment by a Companion Chatbot
• Reinforcement Learning from Human Feedback: Whose Culture, Whose Values, Whose Perspectives?
• Transforming Agency. On the mode of existence of Large Language Models
• Data Poisoning Vulnerabilities Across Healthcare AI Architectures: A Security Threat Analysis
• The Epistemological Consequences of Large Language Models: Rethinking collective intelligence and institutional knowledge
• Projections of Earth's technosphere: Scenario modeling, worldbuilding, and overview of remotely detectable technosignatures
• Plurals: A System for Guiding LLMs Via Simulated Social Ensembles
• Complex QA and language models hybrid architectures, Survey
• Understanding and Mitigating Risks of Generative AI in Financial Services
• Improving the Capabilities of Large Language Model Based Marketing Analytics Copilots With Semantic Search And Fine-Tuning
• Evaluating Language Model Agency through Negotiations
• Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk
• Speaker attribution in German parliamentary debates with QLoRA-adapted large language models
• A Hormetic Approach to the Value-Loading Problem: Preventing the Paperclip Apocalypse?
• Towards AI epidemiology: a measurement standardisation framework for prospective risk detection
• AI of the People, by the People, for the People: A Social Choice Approach to Collective Control of Artificial Intelligence