AI Research Considerations for Human Existential Safety (ARCHES)
arXiv:2006.04948
Abstract
Framed in positive terms, this report examines how technical AI research might be steered in a manner that is more attentive to humanity's long-term prospects for survival as a species. In negative terms, we ask what existential risks humanity might face from AI development in the next century, and by what principles contemporary technical research might be directed to address those risks. A key property of hypothetical AI technologies is introduced, called \emph{prepotence}, which is useful for delineating a variety of potential existential risks from artificial intelligence, even as AI paradigms might shift. A set of \auxref{dirtot} contemporary research \directions are then examined for their potential benefit to existential safety. Each research direction is explained with a scenario-driven motivation, and examples of existing work from which to build. The research directions present their own risks and benefits to society that could occur at various scales of impact, and in particular are not guaranteed to benefit existential safety if major developments in them are deployed without adequate forethought and oversight. As such, each direction is accompanied by a consideration of potentially negative side effects.
References in corpus (38)
- Explaining and Harnessing Adversarial Examples
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- End to End Learning for Self-Driving Cars
- Federated Learning: Strategies for Improving Communication Efficiency
- On Calibration of Modern Neural Networks
- Understanding Neural Networks Through Deep Visualization
- Learning to Communicate with Deep Multi-Agent Reinforcement Learning
- Synthesizing Robust Adversarial Examples
- Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
- Deep reinforcement learning from human preferences
- Learning Confidence for Out-of-Distribution Detection in Neural Networks
- Accurate Uncertainties for Deep Learning Using Calibrated Regression
- Multi-agent Reinforcement Learning in Sequential Social Dilemmas
- Learning Multiagent Communication with Backpropagation
- An approach to reachability analysis for feed-forward ReLU neural networks
- A Dual Approach to Scalable Verification of Deep Networks
- Emergent Complexity via Multi-Agent Competition
- A Deep Hierarchical Approach to Lifelong Learning in Minecraft
- Scalable agent alignment via reward modeling: a research direction
- AI Safety Gridworlds
- Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples
- Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
- Strategic Attentive Writer for Learning Macro-Actions
- Learning End-to-End Goal-Oriented Dialog
- Generating Visual Explanations
- Noise Contrastive Priors for Functional Uncertainty
- Fairness Through Awareness
- Developing Bug-Free Machine Learning Systems With Formal Mathematics
- Federated Control with Hierarchical Multi-Agent Deep Reinforcement Learning
- Agent-Agnostic Human-in-the-Loop Reinforcement Learning
- Penalizing side effects using stepwise relative reachability
- Active Reinforcement Learning: Observing Rewards at a Cost
- Categorizing Variants of Goodhart's Law
- 'Indifference' methods for managing agent rewards
- Literal or Pedagogic Human? Analyzing Human Model Misspecification in Objective Learning
- Scaling shared model governance via model splitting
- A critical look at risk assessments for global catastrophes
- Cause, Responsibility, and Blame: oA Structural-Model Approach
Cited by in corpus (9)
- Worldwide AI Ethics: a review of 200 guidelines and recommendations for AI governance
- Guidelines and Evaluation of Clinical Explainable AI in Medical Image Analysis
- Unsolved Problems in ML Safety
- Open Problems in Cooperative AI
- Negative Human Rights as a Basis for Long-term AI Safety and Regulation
- Truthful AI: Developing and governing AI that does not lie
- On Controllability of AI
- AI Alignment vs. AI Ethical Treatment: 10 Challenges
- Achilles Heels for AGI/ASI via Decision Theoretic Adversaries