Scalable agent alignment via reward modeling: a research direction

Abstract

One obstacle to applying reinforcement learning algorithms to real-world problems is the lack of suitable reward functions. Designing such reward functions is difficult in part because the user only has an implicit understanding of the task objective. This gives rise to the agent alignment problem: how do we create agents that behave in accordance with the user's intentions? We outline a high-level research direction to solve the agent alignment problem centered around reward modeling: learning a reward function from interaction with the user and optimizing the learned reward function with reinforcement learning. We discuss the key challenges we expect to face when scaling reward modeling to complex and general domains, concrete approaches to mitigate these challenges, and ways to establish trust in the resulting agents.

References in corpus (52)

• Continuous control with deep reinforcement learning
• Towards A Rigorous Science of Interpretable Machine Learning
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Empirical Evaluation of Rectified Activations in Convolutional Network
• On Calibration of Modern Neural Networks
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Improved Techniques for Training GANs
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Deep reinforcement learning from human preferences
• Distributed Prioritized Experience Replay
• Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
• On a Formal Model of Safe and Scalable Self-driving Cars
• Deep Learning: A Critical Appraisal
• Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
• Generative Adversarial Imitation Learning
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• A Study on Overfitting in Deep Reinforcement Learning
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• A Note on the Inception Score
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• Thinking Fast and Slow with Deep Learning and Tree Search
• Visualizing and Understanding Atari Agents
• Verifiable Reinforcement Learning via Policy Extraction
• Motivating the Rules of the Game for Adversarial Example Research
• AI Safety Gridworlds
• Constrained Policy Optimization
• Interactive Learning from Policy-Dependent Human Feedback
• Training verified learners with learned verifiers
• Unrestricted Adversarial Examples
• Learning to Understand Goal Specifications by Modelling Reward
• Inverse Reward Design
• Bad Universal Priors and Notions of Optimality
• Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures
• Deep Reinforcement Learning from Policy-Dependent Human Feedback
• AI safety via debate
• Repeated Inverse Reinforcement Learning
• Supervising strong learners by amplifying weak experts
• Goedel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements
• Penalizing side effects using stepwise relative reachability
• Agent-Agnostic Human-in-the-Loop Reinforcement Learning
• A Less Biased Evaluation of Out-of-distribution Sample Detectors
• Online Learning with Gated Linear Networks
• Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic
• Low Impact Artificial Intelligences
• Translating Neuralese
• Logical Induction
• Active Reinforcement Learning with Monte-Carlo Tree Search
• A Formal Solution to the Grain of Truth Problem

Cited by in corpus (48)

• On the Opportunities and Risks of Foundation Models
• Artificial Intelligence, Values and Alignment
• Fine-Tuning Language Models from Human Preferences
• Unsolved Problems in ML Safety
• Learning to Generalize from Sparse and Underspecified Rewards
• Recursively Summarizing Books with Human Feedback
• Defining the scope of AI regulations
• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• Alignment of Language Agents
• Learning to summarize from human feedback
• AI Research Considerations for Human Existential Safety (ARCHES)
• Risks from Learned Optimization in Advanced Machine Learning Systems
• A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges
• Open Problems in Cooperative AI
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
• SafeLife 1.0: Exploring Side Effects in Complex Environments
• Requisite Variety in Ethical Utility Functions for AI Value Alignment
• Learning Human Objectives by Evaluating Hypothetical Behavior
• Modeling AGI Safety Frameworks with Causal Influence Diagrams
• A Human-Centered Approach to Interactive Machine Learning
• Sequential Evaluation and Generation Framework for Combinatorial Recommender System
• Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback
• Inverse Constrained Reinforcement Learning
• Complex QA and language models hybrid architectures, Survey
• Hidden Incentives for Auto-Induced Distributional Shift
• Multi-agent Assessment with QoS Enhancement for HD Map Updates in a Vehicular Network
• Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs
• Deep Inductive Logic Programming meets Reinforcement Learning
• On Controllability of AI
• Parenting: Safe Reinforcement Learning from Human Input
• REALab: An Embedded Perspective on Tampering
• Evolutionary Computation and AI Safety: Research Problems Impeding Routine and Safe Real-world Application of Evolution
• Towards Interactive Reinforcement Learning with Intrinsic Feedback
• Avoiding Tampering Incentives in Deep RL via Decoupled Approval
• The Challenge of Value Alignment: from Fairer Algorithms to AI Safety
• Challenges for Using Impact Regularizers to Avoid Negative Side Effects
• AI Alignment vs. AI Ethical Treatment: 10 Challenges
• Towards Empathic Deep Q-Learning
• Auditing Robot Learning for Safety and Compliance during Deployment
• Deep PQR: Solving Inverse Reinforcement Learning using Anchor Actions
• Learning to Request Guidance in Emergent Communication
• Computational principles of intelligence: learning and reasoning with neural networks
• Value Alignment Verification
• Learning the Value Systems of Agents with Preference-based and Inverse Reinforcement Learning
• Relational Mimic for Visual Adversarial Imitation Learning
• Information Directed Reward Learning for Reinforcement Learning
• MarioMix: Creating Aligned Playstyles for Bots with Interactive Reinforcement Learning
• Generalizing Emergent Communication