AI Safety Gridworlds

Abstract

We present a suite of reinforcement learning environments illustrating various safety properties of intelligent agents. These problems include safe interruptibility, avoiding side effects, absent supervisor, reward gaming, safe exploration, as well as robustness to self-modification, distributional shift, and adversaries. To measure compliance with the intended safe behavior, we equip each environment with a performance function that is hidden from the agent. This allows us to categorize AI safety problems into robustness and specification problems, depending on whether the performance function corresponds to the observed reward function. We evaluate A2C and Rainbow, two recent deep reinforcement learning agents, on our environments and show that they are not able to solve them satisfactorily.

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Explaining and Harnessing Adversarial Examples
• Towards A Rigorous Science of Interpretable Machine Learning
• Noisy Networks for Exploration
• Deep reinforcement learning from human preferences
• Cooperative Inverse Reinforcement Learning
• A Berkeley View of Systems Challenges for AI
• Towards Proving the Adversarial Robustness of Deep Neural Networks
• Constrained Policy Optimization
• Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
• Towards Verified Artificial Intelligence
• Safe Policy Improvement by Minimizing Robust Baseline Regret
• Inverse Reward Design
• Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
• An algorithm with nearly optimal pseudo-regret for both stochastic and adversarial bandits
• Model-based Utility Functions
• Low Impact Artificial Intelligences
• APRIL: Active Preference-learning based Reinforcement Learning
• Sequential Extensions of Causal and Evidential Decision Theory

Cited by in corpus (63)

• Reward Constrained Policy Optimization
• Scalable agent alignment via reward modeling: a research direction
• Assessing Generalization in Deep Reinforcement Learning
• A Lyapunov-based Approach to Safe Reinforcement Learning
• Estimating Risk and Uncertainty in Deep Reinforcement Learning
• Practical Solutions for Machine Learning Safety in Autonomous Vehicles
• Adversarial Machine Learning -- Industry Perspectives
• The Role of Cooperation in Responsible AI Development
• Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences
• Alignment of Language Agents
• Deep Reinforcement Learning for Closed-Loop Blood Glucose Control
• Constrained Upper Confidence Reinforcement Learning
• Trust-Region Method with Deep Reinforcement Learning in Analog Design Space Exploration
• AI Research Considerations for Human Existential Safety (ARCHES)
• Interpretable Multi-Objective Reinforcement Learning through Policy Orchestration
• Reinforcement Learning under Threats
• Penalizing side effects using stepwise relative reachability
• The Faults in Our Pi Stars: Security Issues and Open Challenges in Deep Reinforcement Learning
• Avoiding Side Effects By Considering Future Tasks
• Safety-Guided Deep Reinforcement Learning via Online Gaussian Process Estimation
• Axes for Sociotechnical Inquiry in AI Research
• A Perspective on Objects and Systematic Generalization in Model-Based RL
• Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings
• Model-based Reinforcement Learning from Signal Temporal Logic Specifications
• Constrained episodic reinforcement learning in concave-convex and knapsack settings
• Good and safe uses of AI Oracles
• Constrained Reinforcement Learning Has Zero Duality Gap
• Generalizing from a few environments in safety-critical reinforcement learning
• Improving Safety in Reinforcement Learning Using Model-Based Architectures and Human Intervention
• A Human-Centered Approach to Interactive Machine Learning
• Opponent Aware Reinforcement Learning
• Inverse Constrained Reinforcement Learning
• Causal Analysis of Agent Behavior for AI Safety
• Hidden Incentives for Auto-Induced Distributional Shift
• AGI Agent Safety by Iteratively Improving the Utility Function
• Corrigibility with Utility Preservation
• 'Indifference' methods for managing agent rewards
• RuleMatrix: Visualizing and Understanding Classifiers with Rules
• Safer Deep RL with Shallow MCTS: A Case Study in Pommerman
• Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs
• Modeling Friends and Foes
• Incorporating Deception into CyberBattleSim for Autonomous Defense
• What Would Jiminy Cricket Do? Towards Agents That Behave Morally
• SAT-MARL: Specification Aware Training in Multi-Agent Reinforcement Learning
• Parenting: Safe Reinforcement Learning from Human Input
• TanksWorld: A Multi-Agent Environment for AI Safety Research
• Avoiding Side Effects in Complex Environments
• Evolutionary Computation and AI Safety: Research Problems Impeding Routine and Safe Real-world Application of Evolution
• Categorizing Wireheading in Partially Embedded Agents
• Virtuously Safe Reinforcement Learning
• A Roadmap for Robust End-to-End Alignment
• Learning the Arrow of Time
• Concave Utility Reinforcement Learning with Zero-Constraint Violations
• Learning to Be Cautious
• Defining Admissible Rewards for High Confidence Policy Evaluation
• Don't Forget Your Teacher: A Corrective Reinforcement Learning Framework
• There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning
• Relational Mimic for Visual Adversarial Imitation Learning
• Wield: Systematic Reinforcement Learning With Progressive Randomization
• Inferential Induction: A Novel Framework for Bayesian Reinforcement Learning
• Safe Transformative AI via a Windfall Clause
• Model-Free Risk-Sensitive Reinforcement Learning
• Contributions to Large Scale Bayesian Inference and Adversarial Machine Learning