A Hormetic Approach to the Value-Loading Problem: Preventing the Paperclip Apocalypse?

Abstract

The value-loading problem is a significant challenge for researchers aiming to create artificial intelligence (AI) systems that align with human values and preferences. This problem requires a method to define and regulate safe and optimal limits of AI behaviors. In this work, we propose HALO (Hormetic ALignment via Opponent processes), a regulatory paradigm that uses hormetic analysis to regulate the behavioral patterns of AI. Behavioral hormesis is a phenomenon where low frequencies of a behavior have beneficial effects, while high frequencies are harmful. By modeling behaviors as allostatic opponent processes, we can use either Behavioral Frequency Response Analysis (BFRA) or Behavioral Count Response Analysis (BCRA) to quantify the hormetic limits of repeatable behaviors. We demonstrate how HALO can solve the 'paperclip maximizer' scenario, a thought experiment where an unregulated AI tasked with making paperclips could end up converting all matter in the universe into paperclips. Our approach may be used to help create an evolving database of 'values' based on the hedonic calculus of repeatable behaviors with decreasing marginal utility. This positions HALO as a promising solution for the value-loading problem, which involves embedding human-aligned values into an AI system, and the weak-to-strong generalization problem, which explores whether weak models can supervise stronger models as they become more intelligent. Hence, HALO opens several research avenues that may lead to the development of a computational value system that allows an AI algorithm to learn whether the decisions it makes are right or wrong.

24 pages, 7 figures

References in corpus (16)

• Constitutional AI: Harmlessness from AI Feedback
• Voyager: An Open-Ended Embodied Agent with Large Language Models
• Generative Agents: Interactive Simulacra of Human Behavior
• Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
• Hyperbolic Discounting and Learning over Multiple Horizons
• Measuring Progress on Scalable Oversight for Large Language Models
• Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations
• Beyond Preferences in AI Alignment
• A mathematical model of reward-mediated learning in drug addiction
• Algorithmic Pluralism: A Structural Approach To Equal Opportunity
• A Roadmap to Pluralistic Alignment
• What are human values, and how do we align AI to them?
• Evaluating the Prompt Steerability of Large Language Models
• User-Driven Value Alignment: Understanding Users' Perceptions and Strategies for Addressing Biased and Discriminatory Statements in AI Companions
• Human-Instruction-Free LLM Self-Alignment with Limited Samples
• Self-driven Grounding: Large Language Model Agents with Automatical Language-aligned Skill Learning