Causal Analysis of Agent Behavior for AI Safety
arXiv:2103.03938
Abstract
As machine learning systems become more powerful they also become increasingly unpredictable and opaque. Yet, finding human-understandable explanations of how they work is essential for their safe deployment. This technical report illustrates a methodology for investigating the causal mechanisms that drive the behaviour of artificial agents. Six use cases are covered, each addressing a typical question an analyst might ask about an agent. In particular, we show that each question cannot be addressed by pure observation alone, but instead requires conducting experiments with systematically chosen manipulations so as to generate the correct causal evidence.
arXiv comment16 pages, 16 figures, 6 tables
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Language Models are Few-Shot Learners
- Methods for Interpreting and Understanding Deep Neural Networks
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Learning to reinforcement learn
- Invariant Risk Minimization
- AI Safety Gridworlds
- A Philosophical Treatise of Universal Induction
- Machine Theory of Mind
- Actual causation and the art of modeling
- Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures
- Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings
- Incentives for Responsiveness, Instrumental Control and Impact
- Resolving Spurious Correlations in Causal Models of Environments via Interventions